servererr.txt - Ceph - Ceph

Bug #44247 » servererr.txt

Hu Ye, 02/21/2020 08:58 AM

        This tool won't handle connection error alike things,
       please ensure the proper network environment to test.
        Or ctrl+c when meeting error and restart tests
        using ms-public-type async+rdma
              bind ip:port 172.19.36.252:4567
              worker threads 128
              thinktime(us) 1
 -01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0
 -01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband...
 -01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack...
 -01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack  creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0
 -01-14T08:57:33.272+0800 ffff672f1e80  2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608
 -01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0
 -01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4
 -01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1
 -01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting
 -01-14T08:57:33.272+0800 ffff66af0e80  2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904
 -01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0
 -01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7
 -01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1
 -01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting
 -01-14T08:57:33.272+0800 ffff662efe80  2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200
 -01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0
 -01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10
 -01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1
 -01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting
 -01-14T08:57:33.272+0800 ffffa8c24010 10 --  bind v2:172.19.36.252:4567/0
 -01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
 -01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv Network Stack is not ready for bind yet - postponed
 -01-14T08:57:33.272+0800 ffffa8c24010 10 --  ready
 -01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
 -01-14T08:57:33.272+0800 ffffa8c24010 10  Processor -- bind v2:172.19.36.252:4567/0
 -01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
 -01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband binding_port found active port 1
 -01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 4096 receive buffers
 -01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 1024 send buffers
 -01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init device allow 4194304 completion entries
 -01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
 -01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
 -01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0
 -01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80
 -01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60
 -01-14T08:57:33.328+0800 ffff672f1e80 20  RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567
 -01-14T08:57:33.328+0800 ffffa8c24010 10  Processor -- bind bound to v2:172.19.36.252:4567/0
 -01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0)
 -01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0
 -01-14T08:57:33.328+0800 ffffa8c24010  1  Processor -- start
 -01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
 -01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0
 -01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4
 -01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1
 -01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 start start
 -01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
 -01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
 -01-14T08:57:34.836+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:57:34.836+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17
 -01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00
       tcmalloc: large alloc 1074077696 bytes == 0xaaab152f6000 @  0xffffa8b78750 0xaaaac62e3db4 0xaaaac62ebe88 0xaaaac62e431c 0xaaaac62e4500 0xaaaac62ea5c8 0xaaaac62eb2f8 0xaaaac62ec860 0xaaaac62f4dcc 0xaaaac60396b4 0xaaaac604167c 0xaaaac6047a20 0xffffa8719ed4 0xffffa8bcd088
 -01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096
 -01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:57:35.132+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:57:35.132+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0
 -01-14T08:57:35.136+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:57:35.136+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 17
 -01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1
 -01-14T08:57:35.136+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
 -01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
 -01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084
 -01-14T08:57:35.136+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
 -01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
 -01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
 -01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:57:35.592+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:57:35.596+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:57:35.596+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:57:35.596+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
       hns: error cqe!
 -01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event.
 -01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses.
 -01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR
 -01-14T08:57:45.048+0800 ffff65780e80  1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state  WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8
 -01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800
 -01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096
 -01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00
 -01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
 -01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
 -01-14T08:57:54.004+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:57:54.004+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18
 -01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900
 -01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096
 -01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:57:54.012+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:57:54.012+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:57:54.012+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
 -01-14T08:57:54.012+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075
 -01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 18
 -01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1
 -01-14T08:57:54.012+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
 -01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084
 -01-14T08:57:54.012+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
 -01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
 -01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:58:04.024+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:58:04.228+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:58:04.228+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19
 -01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040
 -01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096
 -01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:04.236+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:58:04.236+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:58:04.236+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:04.236+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075
 -01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 19
 -01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1
 -01-14T08:58:04.236+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084
 -01-14T08:58:04.236+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
 -01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
 -01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:58:14.248+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:58:15.248+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:58:15.248+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20
 -01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0
       tcmalloc: large alloc 2148147200 bytes == 0xaaab553c8000 @  0xffffa8b78750 0xaaaac62e3db4 0xaaaac62ebe88 0xaaaac62e431c 0xaaaac62e4500 0xaaaac62ea5c8 0xaaaac62eb2f8 0xaaaac62ec860 0xaaaac62f4dcc 0xaaaac60396b4 0xaaaac604167c 0xaaaac6047a20 0xffffa8719ed4 0xffffa8bcd088
 -01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096
 -01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:15.816+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:58:15.816+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0
 -01-14T08:58:15.816+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:58:15.816+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 20
 -01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1
 -01-14T08:58:15.816+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084
 -01-14T08:58:15.816+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
 -01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
 -01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:58:25.828+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:58:26.636+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:58:26.636+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21
 -01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40
 -01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096
 -01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:26.640+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:58:26.640+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:26.640+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:26.640+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075
 -01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 21
 -01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1
 -01-14T08:58:26.640+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084
 -01-14T08:58:26.640+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
 -01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
 -01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:58:36.652+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
 -01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:58:36.652+0800 ffff672f1e80  1 -- v2:172.19.36.252:4567/0 reap_dead start
 -01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880
 -01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00
 -01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180
 -01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600
 -01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80
 -01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:58:38.260+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:58:38.260+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22
 -01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0
 -01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096
 -01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:38.264+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:58:38.264+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0
 -01-14T08:58:38.264+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:58:38.264+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 22
 -01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1
 -01-14T08:58:38.264+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084
 -01-14T08:58:38.264+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
 -01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
 -01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:58:48.276+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:58:51.484+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:58:51.484+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23
 -01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0
 -01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096
 -01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:51.492+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
 -01-14T08:58:51.492+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
 -01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0
 -01-14T08:58:51.492+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
 -01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:51.492+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
 -01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075
 -01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
 -01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
 -01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
 -01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
 -01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
 -01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 23
 -01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
 -01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1
 -01-14T08:58:51.492+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
 -01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084
 -01-14T08:58:51.492+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
 -01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
 -01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
 -01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
 -01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
 -01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
 -01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
 -01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
 -01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
 -01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
 -01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
 -01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
 -01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
 -01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
 -01-14T08:59:01.504+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
 -01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
 -01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
 -01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
 -01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
 -01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
 -01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
 -01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
 -01-14T08:59:09.088+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
 -01-14T08:59:09.088+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
 -01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started.
       hr_qp->port_num= 0x1
 -01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24
 -01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280
 -01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768
 -01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768
 -01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers.
 -01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR
 -01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0
 -01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280
       *** Caught signal (Segmentation fault) **
        in thread ffff672f1e80 thread_name:msgr-worker-0
        ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev)
 : (__kernel_rt_sigreturn()+0) [0xffffa8c315c0]
 : (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr<Infiniband>&, std::shared_ptr<RDMADispatcher>&, RDMAWorker*)+0x18c) [0xaaaac62ec874]
 : (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc]
 : (Processor::accept()+0x11c) [0xaaaac60396b4]
 : (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x51c) [0xaaaac604167c]
 : (()+0x469a20) [0xaaaac6047a20]
 : (()+0xc9ed4) [0xffffa8719ed4]
 : (()+0x7088) [0xffffa8bcd088]
 -01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) **
        in thread ffff672f1e80 thread_name:msgr-worker-0
        ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev)
 : (__kernel_rt_sigreturn()+0) [0xffffa8c315c0]
 : (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr<Infiniband>&, std::shared_ptr<RDMADispatcher>&, RDMAWorker*)+0x18c) [0xaaaac62ec874]
 : (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc]
 : (Processor::accept()+0x11c) [0xaaaac60396b4]
 : (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x51c) [0xaaaac604167c]
 : (()+0x469a20) [0xaaaac6047a20]
 : (()+0xc9ed4) [0xffffa8719ed4]
 : (()+0x7088) [0xffffa8bcd088]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- begin dump of recent events ---
         -588> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command assert hook 0xaaab0c49e700
         -587> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command abort hook 0xaaab0c49e700
         -586> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perfcounters_dump hook 0xaaab0c49e700
         -585> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command 1 hook 0xaaab0c49e700
         -584> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf dump hook 0xaaab0c49e700
         -583> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perfcounters_schema hook 0xaaab0c49e700
         -582> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf histogram dump hook 0xaaab0c49e700
         -581> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command 2 hook 0xaaab0c49e700
         -580> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf schema hook 0xaaab0c49e700
         -579> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf histogram schema hook 0xaaab0c49e700
         -578> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf reset hook 0xaaab0c49e700
         -577> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config show hook 0xaaab0c49e700
         -576> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config help hook 0xaaab0c49e700
         -575> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config set hook 0xaaab0c49e700
         -574> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config unset hook 0xaaab0c49e700
         -573> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config get hook 0xaaab0c49e700
         -572> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config diff hook 0xaaab0c49e700
         -571> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config diff get hook 0xaaab0c49e700
         -570> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command injectargs hook 0xaaab0c49e700
         -569> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log flush hook 0xaaab0c49e700
         -568> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log dump hook 0xaaab0c49e700
         -567> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log reopen hook 0xaaab0c49e700
         -566> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command dump_mempools hook 0xaaab0d110068
         -565> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -564> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -563> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -562> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -561> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -560> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -559> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -558> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -557> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -556> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -555> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -554> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -553> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -552> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -551> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -550> 2020-01-14T08:57:33.252+0800 ffffa8c24010  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring
         -549> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0
         -548> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband...
         -547> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack...
         -546> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack  creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0
         -545> 2020-01-14T08:57:33.272+0800 ffff672f1e80  2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608
         -544> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0
         -543> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4
         -542> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1
         -541> 2020-01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting
         -540> 2020-01-14T08:57:33.272+0800 ffff66af0e80  2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904
         -539> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0
         -538> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7
         -537> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1
         -536> 2020-01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting
         -535> 2020-01-14T08:57:33.272+0800 ffff662efe80  2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200
         -534> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0
         -533> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10
         -532> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1
         -531> 2020-01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting
         -530> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -529> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -528> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -527> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -526> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -525> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -524> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -523> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -522> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -521> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -520> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -519> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -518> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -517> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -516> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -515> 2020-01-14T08:57:33.272+0800 ffffa8c24010  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring
         -514> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bind v2:172.19.36.252:4567/0
         -513> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
         -512> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv Network Stack is not ready for bind yet - postponed
         -511> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  ready
         -510> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
         -509> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10  Processor -- bind v2:172.19.36.252:4567/0
         -508> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -507> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband binding_port found active port 1
         -506> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 4096 receive buffers
         -505> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 1024 send buffers
         -504> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init device allow 4194304 completion entries
         -503> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
         -502> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
         -501> 2020-01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0
         -500> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80
         -499> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60
         -498> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20  RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567
         -497> 2020-01-14T08:57:33.328+0800 ffffa8c24010 10  Processor -- bind bound to v2:172.19.36.252:4567/0
         -496> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0)
         -495> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0
         -494> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1  Processor -- start
         -493> 2020-01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -492> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0
         -491> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4
         -490> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1
         -489> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 start start
         -488> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -487> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -486> 2020-01-14T08:57:34.836+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -485> 2020-01-14T08:57:34.836+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -484> 2020-01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started.
         -483> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17
         -482> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00
         -481> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096
         -480> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -479> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -478> 2020-01-14T08:57:35.132+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -477> 2020-01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0
         -476> 2020-01-14T08:57:35.136+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -475> 2020-01-14T08:57:35.136+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -474> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -473> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -472> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -471> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process
         -470> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -469> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -468> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -467> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -466> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -465> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -464> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 17
         -463> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -462> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -461> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1
         -460> 2020-01-14T08:57:35.136+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -459> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -458> 2020-01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084
         -457> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -456> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -455> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -454> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075
         -453> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -452> 2020-01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084
         -451> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -450> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -449> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -448> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8
         -447> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -446> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -445> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -444> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -443> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
         -442> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -441> 2020-01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -440> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -439> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -438> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -437> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -436> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
         -435> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -434> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -433> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -432> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -431> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -430> 2020-01-14T08:57:35.596+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -429> 2020-01-14T08:57:35.596+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -428> 2020-01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -427> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -426> 2020-01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -425> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -424> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -423> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -422> 2020-01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -421> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event.
         -420> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses.
         -419> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR
         -418> 2020-01-14T08:57:45.048+0800 ffff65780e80  1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state  WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8
         -417> 2020-01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800
         -416> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096
         -415> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00
         -414> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -413> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -412> 2020-01-14T08:57:54.004+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -411> 2020-01-14T08:57:54.004+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -410> 2020-01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started.
         -409> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18
         -408> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900
         -407> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096
         -406> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -405> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -404> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -403> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -402> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -401> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0
         -400> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -399> 2020-01-14T08:57:54.012+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -398> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -397> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -396> 2020-01-14T08:57:54.012+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -395> 2020-01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084
         -394> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -393> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -392> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -391> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075
         -390> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process
         -389> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -388> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -387> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -386> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -385> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -384> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -383> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 18
         -382> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -381> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -380> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1
         -379> 2020-01-14T08:57:54.012+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -378> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -377> 2020-01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084
         -376> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -375> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -374> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -373> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8
         -372> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -371> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -370> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -369> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -368> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
         -367> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -366> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -365> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -364> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -363> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -362> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -361> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
         -360> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -359> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -358> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -357> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -356> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -355> 2020-01-14T08:58:04.024+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -354> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -353> 2020-01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -352> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -351> 2020-01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -350> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -349> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -348> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -347> 2020-01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -346> 2020-01-14T08:58:04.228+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -345> 2020-01-14T08:58:04.228+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -344> 2020-01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started.
         -343> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19
         -342> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040
         -341> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096
         -340> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -339> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -338> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -337> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -336> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -335> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0
         -334> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -333> 2020-01-14T08:58:04.236+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -332> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -331> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -330> 2020-01-14T08:58:04.236+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -329> 2020-01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084
         -328> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -327> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -326> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -325> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075
         -324> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process
         -323> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -322> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -321> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -320> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -319> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -318> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -317> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 19
         -316> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -315> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -314> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1
         -313> 2020-01-14T08:58:04.236+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -312> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -311> 2020-01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084
         -310> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -309> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -308> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -307> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0
         -306> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -305> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -304> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -303> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -302> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
         -301> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -300> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -299> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -298> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -297> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -296> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -295> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
         -294> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -293> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -292> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -291> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -290> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -289> 2020-01-14T08:58:14.248+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -288> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -287> 2020-01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -286> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -285> 2020-01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -284> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -283> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -282> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -281> 2020-01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -280> 2020-01-14T08:58:15.248+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -279> 2020-01-14T08:58:15.248+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -278> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started.
         -277> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20
         -276> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0
         -275> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096
         -274> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -273> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -272> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -271> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0
         -270> 2020-01-14T08:58:15.816+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -269> 2020-01-14T08:58:15.816+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -268> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -267> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -266> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -265> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
         -264> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -263> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -262> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -261> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -260> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -259> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -258> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 20
         -257> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -256> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -255> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1
         -254> 2020-01-14T08:58:15.816+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -253> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -252> 2020-01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084
         -251> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -250> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -249> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -248> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075
         -247> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -246> 2020-01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084
         -245> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -244> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -243> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -242> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88
         -241> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -240> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -239> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -238> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -237> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
         -236> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -235> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -234> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -233> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -232> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -231> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -230> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
         -229> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -228> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -227> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -226> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -225> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -224> 2020-01-14T08:58:25.828+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -223> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -222> 2020-01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -221> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -220> 2020-01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -219> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -218> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -217> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -216> 2020-01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -215> 2020-01-14T08:58:26.636+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -214> 2020-01-14T08:58:26.636+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -213> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started.
         -212> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21
         -211> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40
         -210> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096
         -209> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -208> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -207> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -206> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -205> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -204> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -203> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0
         -202> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -201> 2020-01-14T08:58:26.640+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -200> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -199> 2020-01-14T08:58:26.640+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -198> 2020-01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084
         -197> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -196> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -195> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -194> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075
         -193> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process
         -192> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -191> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -190> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -189> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -188> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -187> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -186> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 21
         -185> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -184> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -183> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1
         -182> 2020-01-14T08:58:26.640+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -181> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -180> 2020-01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084
         -179> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -178> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -177> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -176> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60
         -175> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -174> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -173> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -172> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -171> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
         -170> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -169> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -168> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -167> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -166> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -165> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -164> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
         -163> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -162> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -161> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -160> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -159> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -158> 2020-01-14T08:58:36.652+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -157> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -156> 2020-01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -155> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -154> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -153> 2020-01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -152> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -151> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -150> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -149> 2020-01-14T08:58:36.652+0800 ffff672f1e80  1 -- v2:172.19.36.252:4567/0 reap_dead start
         -148> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880
         -147> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00
         -146> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180
         -145> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600
         -144> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80
         -143> 2020-01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -142> 2020-01-14T08:58:38.260+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -141> 2020-01-14T08:58:38.260+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -140> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started.
         -139> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22
         -138> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0
         -137> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096
         -136> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -135> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -134> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -133> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0
         -132> 2020-01-14T08:58:38.264+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -131> 2020-01-14T08:58:38.264+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -130> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -129> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -128> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -127> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process
         -126> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -125> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -124> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -123> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -122> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -121> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -120> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 22
         -119> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -118> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -117> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1
         -116> 2020-01-14T08:58:38.264+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -115> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
         -114> 2020-01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084
         -113> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -112> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -111> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -110> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075
         -109> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
         -108> 2020-01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084
         -107> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -106> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -105> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -104> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38
         -103> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -102> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -101> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -100> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -99> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
          -98> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
          -97> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
          -96> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
          -95> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
          -94> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -93> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -92> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
          -91> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
          -90> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
          -89> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
          -88> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
          -87> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
          -86> 2020-01-14T08:58:48.276+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
          -85> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
          -84> 2020-01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
          -83> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
          -82> 2020-01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
          -81> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
          -80> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
          -79> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
          -78> 2020-01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
          -77> 2020-01-14T08:58:51.484+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
          -76> 2020-01-14T08:58:51.484+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
          -75> 2020-01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started.
          -74> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23
          -73> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0
          -72> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096
          -71> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
          -70> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
          -69> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
          -68> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
          -67> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
          -66> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0
          -65> 2020-01-14T08:58:51.492+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
          -64> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
          -63> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
          -62> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -61> 2020-01-14T08:58:51.492+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
          -60> 2020-01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084
          -59> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
          -58> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
          -57> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
          -56> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075
          -55> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
          -54> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
          -53> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
          -52> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
          -51> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
          -50> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
          -49> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
          -48> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 23
          -47> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
          -46> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
          -45> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1
          -44> 2020-01-14T08:58:51.492+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
          -43> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -42> 2020-01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084
          -41> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
          -40> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
          -39> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
          -38> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10
          -37> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
          -36> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
          -35> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -34> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -33> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
          -32> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -31> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
          -30> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
          -29> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
          -28> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -27> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -26> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
          -25> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
          -24> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
          -23> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
          -22> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
          -21> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
          -20> 2020-01-14T08:59:01.504+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
          -19> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
          -18> 2020-01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
          -17> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
          -16> 2020-01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
          -15> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
          -14> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
          -13> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
          -12> 2020-01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
          -11> 2020-01-14T08:59:09.088+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
          -10> 2020-01-14T08:59:09.088+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
           -9> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started.
           -8> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24
           -7> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280
           -6> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768
           -5> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768
           -4> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers.
           -3> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR
           -2> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0
           -1> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280
 > 2020-01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) **
        in thread ffff672f1e80 thread_name:msgr-worker-0
        ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev)
 : (__kernel_rt_sigreturn()+0) [0xffffa8c315c0]
 : (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr<Infiniband>&, std::shared_ptr<RDMADispatcher>&, RDMAWorker*)+0x18c) [0xaaaac62ec874]
 : (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc]
 : (Processor::accept()+0x11c) [0xaaaac60396b4]
 : (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x51c) [0xaaaac604167c]
 : (()+0x469a20) [0xaaaac6047a20]
 : (()+0xc9ed4) [0xffffa8719ed4]
 : (()+0x7088) [0xffffa8bcd088]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- logging levels ---
 / 5 none
 / 1 lockdep
 / 1 context
 / 1 crush
 / 5 mds
 / 5 mds_balancer
 / 5 mds_locker
 / 5 mds_log
 / 5 mds_log_expire
 / 5 mds_migrator
 / 1 buffer
 / 1 timer
 / 1 filer
 / 1 striper
 / 1 objecter
 / 5 rados
 / 5 rbd
 / 5 rbd_mirror
 / 5 rbd_replay
 / 5 journaler
 / 5 objectcacher
 / 5 immutable_obj_cache
 / 5 client
 / 5 osd
 / 5 optracker
 / 5 objclass
 / 3 filestore
 / 3 journal
 /20 ms
 / 5 mon
 /10 monc
 / 5 paxos
 / 5 tp
 / 5 auth
 / 5 crypto
 / 1 finisher
 / 1 reserver
 / 5 heartbeatmap
 / 5 perfcounter
 / 5 rgw
 / 5 rgw_sync
 /10 civetweb
 / 5 javaclient
 / 5 asok
 / 1 throttle
 / 0 refs
 / 5 compressor
 / 5 bluestore
 / 5 bluefs
 / 3 bdev
 / 5 kstore
 / 5 rocksdb
 / 5 leveldb
 / 5 memdb
 / 5 fuse
 / 5 mgr
 / 5 mgrc
 / 5 dpdk
 / 5 eventtrace
 / 5 prioritycache
 / 5 test
         -2/-2 (syslog threshold)
 /99 (stderr threshold)
         max_recent       500
         max_new         1000
         log_file
       --- end dump of recent events ---
       --- begin dump of recent events ---
         -588> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command assert hook 0xaaab0c49e700
         -587> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command abort hook 0xaaab0c49e700
         -586> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perfcounters_dump hook 0xaaab0c49e700
         -585> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command 1 hook 0xaaab0c49e700
         -584> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf dump hook 0xaaab0c49e700
         -583> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perfcounters_schema hook 0xaaab0c49e700
         -582> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf histogram dump hook 0xaaab0c49e700
         -581> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command 2 hook 0xaaab0c49e700
         -580> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf schema hook 0xaaab0c49e700
         -579> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf histogram schema hook 0xaaab0c49e700
         -578> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command perf reset hook 0xaaab0c49e700
         -577> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config show hook 0xaaab0c49e700
         -576> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config help hook 0xaaab0c49e700
         -575> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config set hook 0xaaab0c49e700
         -574> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config unset hook 0xaaab0c49e700
         -573> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config get hook 0xaaab0c49e700
         -572> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config diff hook 0xaaab0c49e700
         -571> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command config diff get hook 0xaaab0c49e700
         -570> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command injectargs hook 0xaaab0c49e700
         -569> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log flush hook 0xaaab0c49e700
         -568> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log dump hook 0xaaab0c49e700
         -567> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command log reopen hook 0xaaab0c49e700
         -566> 2020-01-14T08:57:33.240+0800 ffffa8c24010  5 asok(0xaaab0c51e000) register_command dump_mempools hook 0xaaab0d110068
         -565> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -564> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -563> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx
         -562> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -561> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -560> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -559> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -558> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -557> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -556> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -555> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -554> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -553> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -552> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: crc
         -551> 2020-01-14T08:57:33.252+0800 ffffa8c24010  5 AuthRegistry(0xaaab0c592148) adding con mode: secure
         -550> 2020-01-14T08:57:33.252+0800 ffffa8c24010  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring
         -549> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0
         -548> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband...
         -547> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack...
         -546> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack  creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0
         -545> 2020-01-14T08:57:33.272+0800 ffff672f1e80  2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608
         -544> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0
         -543> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4
         -542> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1
         -541> 2020-01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting
         -540> 2020-01-14T08:57:33.272+0800 ffff66af0e80  2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904
         -539> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0
         -538> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7
         -537> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1
         -536> 2020-01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting
         -535> 2020-01-14T08:57:33.272+0800 ffff662efe80  2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200
         -534> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0
         -533> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10
         -532> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1
         -531> 2020-01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting
         -530> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -529> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -528> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx
         -527> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -526> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -525> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -524> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -523> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -522> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -521> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -520> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -519> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -518> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -517> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: crc
         -516> 2020-01-14T08:57:33.272+0800 ffffa8c24010  5 AuthRegistry(0xfffffeb401b8) adding con mode: secure
         -515> 2020-01-14T08:57:33.272+0800 ffffa8c24010  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring
         -514> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bind v2:172.19.36.252:4567/0
         -513> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
         -512> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv Network Stack is not ready for bind yet - postponed
         -511> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  ready
         -510> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 --  bindv v2:172.19.36.252:4567/0
         -509> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10  Processor -- bind v2:172.19.36.252:4567/0
         -508> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -507> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband binding_port found active port 1
         -506> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 4096 receive buffers
         -505> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init assigning: 1024 send buffers
         -504> 2020-01-14T08:57:33.280+0800 ffff672f1e80  1 Infiniband init device allow 4194304 completion entries
         -503> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
         -502> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started.
         -501> 2020-01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0
         -500> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80
         -499> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60
         -498> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20  RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567
         -497> 2020-01-14T08:57:33.328+0800 ffffa8c24010 10  Processor -- bind bound to v2:172.19.36.252:4567/0
         -496> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0)
         -495> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0
         -494> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1  Processor -- start
         -493> 2020-01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -492> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0
         -491> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4
         -490> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1
         -489> 2020-01-14T08:57:33.328+0800 ffffa8c24010  1 -- v2:172.19.36.252:4567/0 start start
         -488> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -487> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -486> 2020-01-14T08:57:34.836+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -485> 2020-01-14T08:57:34.836+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -484> 2020-01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started.
         -483> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17
         -482> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00
         -481> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096
         -480> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -479> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -478> 2020-01-14T08:57:35.132+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -477> 2020-01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0
         -476> 2020-01-14T08:57:35.136+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -475> 2020-01-14T08:57:35.136+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -474> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -473> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -472> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -471> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process
         -470> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -469> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -468> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -467> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -466> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -465> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -464> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 17
         -463> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -462> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -461> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1
         -460> 2020-01-14T08:57:35.136+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -459> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -458> 2020-01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084
         -457> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -456> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -455> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -454> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075
         -453> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -452> 2020-01-14T08:57:35.136+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084
         -451> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -450> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -449> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -448> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8
         -447> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -446> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -445> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -444> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -443> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
         -442> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29
         -441> 2020-01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -440> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -439> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -438> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -437> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -436> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0
         -435> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -434> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -433> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -432> 2020-01-14T08:57:35.592+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -431> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -430> 2020-01-14T08:57:35.596+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -429> 2020-01-14T08:57:35.596+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -428> 2020-01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -427> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -426> 2020-01-14T08:57:35.596+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -425> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -424> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -423> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -422> 2020-01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -421> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event.
         -420> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses.
         -419> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR
         -418> 2020-01-14T08:57:45.048+0800 ffff65780e80  1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state  WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8
         -417> 2020-01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800
         -416> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096
         -415> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00
         -414> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -413> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started.
         -412> 2020-01-14T08:57:54.004+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -411> 2020-01-14T08:57:54.004+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -410> 2020-01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started.
         -409> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18
         -408> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900
         -407> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096
         -406> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -405> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -404> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -403> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -402> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -401> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0
         -400> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -399> 2020-01-14T08:57:54.012+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -398> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -397> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -396> 2020-01-14T08:57:54.012+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -395> 2020-01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084
         -394> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -393> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -392> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -391> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075
         -390> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process
         -389> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -388> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -387> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -386> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -385> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -384> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -383> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 18
         -382> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -381> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -380> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1
         -379> 2020-01-14T08:57:54.012+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -378> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -377> 2020-01-14T08:57:54.012+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084
         -376> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -375> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -374> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -373> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8
         -372> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -371> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -370> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -369> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -368> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
         -367> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29
         -366> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -365> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -364> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -363> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -362> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -361> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0
         -360> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -359> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -358> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -357> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -356> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -355> 2020-01-14T08:58:04.024+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -354> 2020-01-14T08:58:04.024+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -353> 2020-01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -352> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -351> 2020-01-14T08:58:04.024+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -350> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -349> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -348> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -347> 2020-01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -346> 2020-01-14T08:58:04.228+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -345> 2020-01-14T08:58:04.228+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -344> 2020-01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started.
         -343> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19
         -342> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040
         -341> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096
         -340> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -339> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -338> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -337> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -336> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -335> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0
         -334> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -333> 2020-01-14T08:58:04.236+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -332> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -331> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -330> 2020-01-14T08:58:04.236+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -329> 2020-01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084
         -328> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -327> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -326> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -325> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075
         -324> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process
         -323> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -322> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -321> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -320> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -319> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -318> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -317> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 19
         -316> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -315> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -314> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1
         -313> 2020-01-14T08:58:04.236+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -312> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -311> 2020-01-14T08:58:04.236+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084
         -310> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -309> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -308> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -307> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0
         -306> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -305> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -304> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -303> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -302> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
         -301> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29
         -300> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -299> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -298> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -297> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -296> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -295> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0
         -294> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -293> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -292> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -291> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -290> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -289> 2020-01-14T08:58:14.248+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -288> 2020-01-14T08:58:14.248+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -287> 2020-01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -286> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -285> 2020-01-14T08:58:14.248+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -284> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -283> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -282> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -281> 2020-01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -280> 2020-01-14T08:58:15.248+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -279> 2020-01-14T08:58:15.248+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -278> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started.
         -277> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20
         -276> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0
         -275> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096
         -274> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -273> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -272> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -271> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0
         -270> 2020-01-14T08:58:15.816+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -269> 2020-01-14T08:58:15.816+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -268> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -267> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -266> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -265> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
         -264> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -263> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -262> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -261> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -260> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -259> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -258> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 20
         -257> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -256> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -255> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1
         -254> 2020-01-14T08:58:15.816+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -253> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -252> 2020-01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084
         -251> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -250> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -249> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -248> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075
         -247> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -246> 2020-01-14T08:58:15.816+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084
         -245> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -244> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -243> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -242> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88
         -241> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -240> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -239> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -238> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -237> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
         -236> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29
         -235> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -234> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -233> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -232> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -231> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -230> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0
         -229> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -228> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -227> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -226> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -225> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -224> 2020-01-14T08:58:25.828+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -223> 2020-01-14T08:58:25.828+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -222> 2020-01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -221> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -220> 2020-01-14T08:58:25.828+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -219> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -218> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -217> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -216> 2020-01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -215> 2020-01-14T08:58:26.636+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -214> 2020-01-14T08:58:26.636+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -213> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started.
         -212> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21
         -211> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40
         -210> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096
         -209> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -208> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -207> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -206> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -205> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -204> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -203> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0
         -202> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -201> 2020-01-14T08:58:26.640+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -200> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -199> 2020-01-14T08:58:26.640+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -198> 2020-01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084
         -197> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -196> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -195> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -194> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075
         -193> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process
         -192> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -191> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -190> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -189> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -188> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -187> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -186> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 21
         -185> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -184> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -183> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1
         -182> 2020-01-14T08:58:26.640+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
         -181> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -180> 2020-01-14T08:58:26.640+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084
         -179> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -178> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -177> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -176> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60
         -175> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -174> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -173> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -172> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -171> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
         -170> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29
         -169> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
         -168> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
         -167> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
         -166> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -165> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
         -164> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0
         -163> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
         -162> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
         -161> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
         -160> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
         -159> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
         -158> 2020-01-14T08:58:36.652+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
         -157> 2020-01-14T08:58:36.652+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
         -156> 2020-01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
         -155> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
         -154> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup
         -153> 2020-01-14T08:58:36.652+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
         -152> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
         -151> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
         -150> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
         -149> 2020-01-14T08:58:36.652+0800 ffff672f1e80  1 -- v2:172.19.36.252:4567/0 reap_dead start
         -148> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880
         -147> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00
         -146> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180
         -145> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600
         -144> 2020-01-14T08:58:36.652+0800 ffff672f1e80  5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80
         -143> 2020-01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
         -142> 2020-01-14T08:58:38.260+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
         -141> 2020-01-14T08:58:38.260+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -140> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started.
         -139> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22
         -138> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0
         -137> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096
         -136> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
         -135> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
         -134> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
         -133> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0
         -132> 2020-01-14T08:58:38.264+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
         -131> 2020-01-14T08:58:38.264+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
         -130> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
         -129> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
         -128> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
         -127> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process
         -126> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
         -125> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
         -124> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
         -123> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
         -122> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
         -121> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
         -120> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 22
         -119> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
         -118> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
         -117> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1
         -116> 2020-01-14T08:58:38.264+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not active. len: 4096
         -115> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
         -114> 2020-01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084
         -113> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
         -112> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
         -111> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
         -110> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075
         -109> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
         -108> 2020-01-14T08:58:38.264+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084
         -107> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
         -106> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
         -105> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
         -104> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38
         -103> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
         -102> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
         -101> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
         -100> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -99> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
          -98> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29
          -97> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
          -96> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
          -95> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
          -94> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -93> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -92> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0
          -91> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
          -90> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
          -89> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
          -88> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
          -87> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
          -86> 2020-01-14T08:58:48.276+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
          -85> 2020-01-14T08:58:48.276+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
          -84> 2020-01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
          -83> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
          -82> 2020-01-14T08:58:48.276+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
          -81> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
          -80> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
          -79> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
          -78> 2020-01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
          -77> 2020-01-14T08:58:51.484+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
          -76> 2020-01-14T08:58:51.484+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
          -75> 2020-01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started.
          -74> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23
          -73> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0
          -72> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096
          -71> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
          -70> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28
          -69> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10  Processor -- accept accepted incoming on sd 29
          -68> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
          -67> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7
          -66> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0
          -65> 2020-01-14T08:58:51.492+0800 ffff672f1e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept
          -64> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
          -63> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup
          -62> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -61> 2020-01-14T08:58:51.492+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
          -60> 2020-01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084
          -59> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3
          -58> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully.
          -57> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully.
          -56> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075
          -55> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process
          -54> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0
          -53> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7
          -52> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1
          -51> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event
          -50> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange
          -49> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange
          -48> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl send fake send to upper, QP: 23
          -47> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0
          -46> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10
          -45> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1
          -44> 2020-01-14T08:58:51.492+0800 ffff66af0e80  1  RDMAConnectedSocketImpl read when ib not connected. len: 4096
          -43> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -42> 2020-01-14T08:58:51.492+0800 ffff66af0e80  5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084
          -41> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10  RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1
          -40> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2
          -39> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
          -38> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10
          -37> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
          -36> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl submit finished sending 26 bytes.
          -35> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -34> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -33> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
          -32> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29
          -31> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message
          -30> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl handle_connection recv handshake msg failed.
          -29> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1  RDMAConnectedSocketImpl fault tcp fd 28
          -28> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process
          -27> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10
          -26> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0
          -25> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104
          -24> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 -- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
          -23> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1
          -22> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted)
          -21> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault
          -20> 2020-01-14T08:59:01.504+0800 ffff66af0e80  2 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half  accept state just closed
          -19> 2020-01-14T08:59:01.504+0800 ffff66af0e80  1 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
          -18> 2020-01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state
          -17> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started
          -16> 2020-01-14T08:59:01.504+0800 ffff66af0e80  5 --2- v2:172.19.36.252:4567/0 >>  conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers
          -15> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7
          -14> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
          -13> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7
          -12> 2020-01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument
          -11> 2020-01-14T08:59:09.088+0800 ffff672f1e80 10  Processor -- accept listen_fd=27
          -10> 2020-01-14T08:59:09.088+0800 ffff672f1e80 15  RDMAServerSocketImpl accept
           -9> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started.
           -8> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24
           -7> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280
           -6> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768
           -5> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768
           -4> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers.
           -3> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR
           -2> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0
           -1> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280
 > 2020-01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) **
        in thread ffff672f1e80 thread_name:msgr-worker-0
        ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev)
 : (__kernel_rt_sigreturn()+0) [0xffffa8c315c0]
 : (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr<Infiniband>&, std::shared_ptr<RDMADispatcher>&, RDMAWorker*)+0x18c) [0xaaaac62ec874]
 : (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc]
 : (Processor::accept()+0x11c) [0xaaaac60396b4]
 : (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x51c) [0xaaaac604167c]
 : (()+0x469a20) [0xaaaac6047a20]
 : (()+0xc9ed4) [0xffffa8719ed4]
 : (()+0x7088) [0xffffa8bcd088]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- logging levels ---
 / 5 none
 / 1 lockdep
 / 1 context
 / 1 crush
 / 5 mds
 / 5 mds_balancer
 / 5 mds_locker
 / 5 mds_log
 / 5 mds_log_expire
 / 5 mds_migrator
 / 1 buffer
 / 1 timer
 / 1 filer
 / 1 striper
 / 1 objecter
 / 5 rados
 / 5 rbd
 / 5 rbd_mirror
 / 5 rbd_replay
 / 5 journaler
 / 5 objectcacher
 / 5 immutable_obj_cache
 / 5 client
 / 5 osd
 / 5 optracker
 / 5 objclass
 / 3 filestore
 / 3 journal
 /20 ms
 / 5 mon
 /10 monc
 / 5 paxos
 / 5 tp
 / 5 auth
 / 5 crypto
 / 1 finisher
 / 1 reserver
 / 5 heartbeatmap
 / 5 perfcounter
 / 5 rgw
 / 5 rgw_sync
 /10 civetweb
 / 5 javaclient
 / 5 asok
 / 1 throttle
 / 0 refs
 / 5 compressor
 / 5 bluestore
 / 5 bluefs
 / 3 bdev
 / 5 kstore
 / 5 rocksdb
 / 5 leveldb
 / 5 memdb
 / 5 fuse
 / 5 mgr
 / 5 mgrc
 / 5 dpdk
 / 5 eventtrace
 / 5 prioritycache
 / 5 test
         -2/-2 (syslog threshold)
 /99 (stderr threshold)
         max_recent       500
         max_new         1000
         log_file /var/lib/ceph/crash/2020-01-14T00:59:09.101446Z_5af684cd-6ecb-4f45-8495-307fc79fc807/log
       --- end dump of recent events ---
       Segmentation fault (core dumped)
       root@node2:~/testceph#
       root@node2:~/testceph#
       root@node2:~/testceph#
       root@node2:~/testceph#

(1-1/1)

Project

General

Profile

Ceph

Bug #44247 » servererr.txt