This tool won't handle connection error alike things, please ensure the proper network environment to test. Or ctrl+c when meeting error and restart tests using ms-public-type async+rdma bind ip:port 172.19.36.252:4567 worker threads 128 thinktime(us) 1 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband... 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack... 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0 2020-01-14T08:57:33.272+0800 ffff672f1e80 2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1 2020-01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting 2020-01-14T08:57:33.272+0800 ffff66af0e80 2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1 2020-01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting 2020-01-14T08:57:33.272+0800 ffff662efe80 2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0 2020-01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1 2020-01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bind v2:172.19.36.252:4567/0 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv Network Stack is not ready for bind yet - postponed 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- ready 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 Processor -- bind v2:172.19.36.252:4567/0 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband binding_port found active port 1 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 4096 receive buffers 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 1024 send buffers 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init device allow 4194304 completion entries 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. 2020-01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80 2020-01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567 2020-01-14T08:57:33.328+0800 ffffa8c24010 10 Processor -- bind bound to v2:172.19.36.252:4567/0 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0) 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 Processor -- start 2020-01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 start start 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. 2020-01-14T08:57:34.836+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:57:34.836+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00 tcmalloc: large alloc 1074077696 bytes == 0xaaab152f6000 @ 0xffffa8b78750 0xaaaac62e3db4 0xaaaac62ebe88 0xaaaac62e431c 0xaaaac62e4500 0xaaaac62ea5c8 0xaaaac62eb2f8 0xaaaac62ec860 0xaaaac62f4dcc 0xaaaac60396b4 0xaaaac604167c 0xaaaac6047a20 0xffffa8719ed4 0xffffa8bcd088 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:57:35.132+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0 2020-01-14T08:57:35.136+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:57:35.136+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 17 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1 2020-01-14T08:57:35.136+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 2020-01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:57:35.596+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:57:35.596+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument hns: error cqe! 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event. 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses. 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR 2020-01-14T08:57:45.048+0800 ffff65780e80 1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8 2020-01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. 2020-01-14T08:57:54.004+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:57:54.004+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:57:54.012+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 2020-01-14T08:57:54.012+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 18 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1 2020-01-14T08:57:54.012+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:58:04.024+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:58:04.228+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:58:04.228+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:58:04.236+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:04.236+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 19 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1 2020-01-14T08:58:04.236+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:58:14.248+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:58:15.248+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:58:15.248+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0 tcmalloc: large alloc 2148147200 bytes == 0xaaab553c8000 @ 0xffffa8b78750 0xaaaac62e3db4 0xaaaac62ebe88 0xaaaac62e431c 0xaaaac62e4500 0xaaaac62ea5c8 0xaaaac62eb2f8 0xaaaac62ec860 0xaaaac62f4dcc 0xaaaac60396b4 0xaaaac604167c 0xaaaac6047a20 0xffffa8719ed4 0xffffa8bcd088 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0 2020-01-14T08:58:15.816+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:58:15.816+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 20 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1 2020-01-14T08:58:15.816+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:58:25.828+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:58:26.636+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:58:26.636+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:26.640+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:26.640+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 21 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1 2020-01-14T08:58:26.640+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:58:36.652+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:58:36.652+0800 ffff672f1e80 1 -- v2:172.19.36.252:4567/0 reap_dead start 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80 2020-01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:58:38.260+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:58:38.260+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0 2020-01-14T08:58:38.264+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:58:38.264+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 22 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1 2020-01-14T08:58:38.264+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:58:48.276+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:58:51.484+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:58:51.484+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0 2020-01-14T08:58:51.492+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:51.492+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 23 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1 2020-01-14T08:58:51.492+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault 2020-01-14T08:59:01.504+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 2020-01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument 2020-01-14T08:59:09.088+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 2020-01-14T08:59:09.088+0800 ffff672f1e80 15 RDMAServerSocketImpl accept 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started. hr_qp->port_num= 0x1 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers. 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280 *** Caught signal (Segmentation fault) ** in thread ffff672f1e80 thread_name:msgr-worker-0 ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev) 1: (__kernel_rt_sigreturn()+0) [0xffffa8c315c0] 2: (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr&, std::shared_ptr&, RDMAWorker*)+0x18c) [0xaaaac62ec874] 3: (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc] 4: (Processor::accept()+0x11c) [0xaaaac60396b4] 5: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x51c) [0xaaaac604167c] 6: (()+0x469a20) [0xaaaac6047a20] 7: (()+0xc9ed4) [0xffffa8719ed4] 8: (()+0x7088) [0xffffa8bcd088] 2020-01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) ** in thread ffff672f1e80 thread_name:msgr-worker-0 ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev) 1: (__kernel_rt_sigreturn()+0) [0xffffa8c315c0] 2: (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr&, std::shared_ptr&, RDMAWorker*)+0x18c) [0xaaaac62ec874] 3: (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc] 4: (Processor::accept()+0x11c) [0xaaaac60396b4] 5: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x51c) [0xaaaac604167c] 6: (()+0x469a20) [0xaaaac6047a20] 7: (()+0xc9ed4) [0xffffa8719ed4] 8: (()+0x7088) [0xffffa8bcd088] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -588> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command assert hook 0xaaab0c49e700 -587> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command abort hook 0xaaab0c49e700 -586> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perfcounters_dump hook 0xaaab0c49e700 -585> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command 1 hook 0xaaab0c49e700 -584> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf dump hook 0xaaab0c49e700 -583> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perfcounters_schema hook 0xaaab0c49e700 -582> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf histogram dump hook 0xaaab0c49e700 -581> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command 2 hook 0xaaab0c49e700 -580> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf schema hook 0xaaab0c49e700 -579> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf histogram schema hook 0xaaab0c49e700 -578> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf reset hook 0xaaab0c49e700 -577> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config show hook 0xaaab0c49e700 -576> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config help hook 0xaaab0c49e700 -575> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config set hook 0xaaab0c49e700 -574> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config unset hook 0xaaab0c49e700 -573> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config get hook 0xaaab0c49e700 -572> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config diff hook 0xaaab0c49e700 -571> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config diff get hook 0xaaab0c49e700 -570> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command injectargs hook 0xaaab0c49e700 -569> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log flush hook 0xaaab0c49e700 -568> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log dump hook 0xaaab0c49e700 -567> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log reopen hook 0xaaab0c49e700 -566> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command dump_mempools hook 0xaaab0d110068 -565> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -564> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -563> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -562> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -561> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -560> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -559> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -558> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -557> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -556> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -555> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -554> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -553> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -552> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -551> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -550> 2020-01-14T08:57:33.252+0800 ffffa8c24010 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -549> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0 -548> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband... -547> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack... -546> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0 -545> 2020-01-14T08:57:33.272+0800 ffff672f1e80 2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608 -544> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0 -543> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4 -542> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1 -541> 2020-01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting -540> 2020-01-14T08:57:33.272+0800 ffff66af0e80 2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904 -539> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0 -538> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7 -537> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1 -536> 2020-01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting -535> 2020-01-14T08:57:33.272+0800 ffff662efe80 2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200 -534> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0 -533> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10 -532> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1 -531> 2020-01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting -530> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -529> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -528> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -527> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -526> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -525> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -524> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -523> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -522> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -521> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -520> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -519> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -518> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -517> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -516> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -515> 2020-01-14T08:57:33.272+0800 ffffa8c24010 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -514> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bind v2:172.19.36.252:4567/0 -513> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 -512> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv Network Stack is not ready for bind yet - postponed -511> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- ready -510> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 -509> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 Processor -- bind v2:172.19.36.252:4567/0 -508> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -507> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband binding_port found active port 1 -506> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 4096 receive buffers -505> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 1024 send buffers -504> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init device allow 4194304 completion entries -503> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. -502> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. -501> 2020-01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0 -500> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80 -499> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60 -498> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567 -497> 2020-01-14T08:57:33.328+0800 ffffa8c24010 10 Processor -- bind bound to v2:172.19.36.252:4567/0 -496> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0) -495> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0 -494> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 Processor -- start -493> 2020-01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -492> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0 -491> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4 -490> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1 -489> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 start start -488> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. -487> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. -486> 2020-01-14T08:57:34.836+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -485> 2020-01-14T08:57:34.836+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -484> 2020-01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started. -483> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17 -482> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00 -481> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096 -480> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -479> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -478> 2020-01-14T08:57:35.132+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -477> 2020-01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0 -476> 2020-01-14T08:57:35.136+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -475> 2020-01-14T08:57:35.136+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -474> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -473> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -472> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -471> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process -470> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -469> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -468> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -467> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -466> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -465> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -464> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 17 -463> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -462> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -461> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1 -460> 2020-01-14T08:57:35.136+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -459> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -458> 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084 -457> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -456> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -455> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -454> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075 -453> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -452> 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084 -451> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -450> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -449> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -448> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8 -447> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -446> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -445> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -444> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -443> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 -442> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -441> 2020-01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -440> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -439> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -438> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -437> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -436> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 -435> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -434> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -433> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -432> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -431> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -430> 2020-01-14T08:57:35.596+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -429> 2020-01-14T08:57:35.596+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -428> 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -427> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -426> 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -425> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -424> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -423> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -422> 2020-01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -421> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event. -420> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses. -419> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR -418> 2020-01-14T08:57:45.048+0800 ffff65780e80 1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8 -417> 2020-01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800 -416> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096 -415> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00 -414> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. -413> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. -412> 2020-01-14T08:57:54.004+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -411> 2020-01-14T08:57:54.004+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -410> 2020-01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started. -409> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18 -408> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900 -407> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096 -406> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -405> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -404> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -403> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -402> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -401> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0 -400> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -399> 2020-01-14T08:57:54.012+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -398> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -397> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -396> 2020-01-14T08:57:54.012+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -395> 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084 -394> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -393> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -392> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -391> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075 -390> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process -389> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -388> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -387> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -386> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -385> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -384> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -383> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 18 -382> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -381> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -380> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1 -379> 2020-01-14T08:57:54.012+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -378> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -377> 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084 -376> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -375> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -374> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -373> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8 -372> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -371> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -370> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -369> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -368> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 -367> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -366> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -365> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -364> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -363> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -362> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -361> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 -360> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -359> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -358> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -357> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -356> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -355> 2020-01-14T08:58:04.024+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -354> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -353> 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -352> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -351> 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -350> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -349> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -348> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -347> 2020-01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -346> 2020-01-14T08:58:04.228+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -345> 2020-01-14T08:58:04.228+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -344> 2020-01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started. -343> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19 -342> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040 -341> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096 -340> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -339> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -338> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -337> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -336> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -335> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0 -334> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -333> 2020-01-14T08:58:04.236+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -332> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -331> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -330> 2020-01-14T08:58:04.236+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -329> 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084 -328> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -327> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -326> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -325> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075 -324> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process -323> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -322> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -321> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -320> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -319> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -318> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -317> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 19 -316> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -315> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -314> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1 -313> 2020-01-14T08:58:04.236+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -312> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -311> 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084 -310> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -309> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -308> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -307> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0 -306> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -305> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -304> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -303> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -302> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 -301> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -300> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -299> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -298> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -297> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -296> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -295> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 -294> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -293> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -292> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -291> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -290> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -289> 2020-01-14T08:58:14.248+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -288> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -287> 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -286> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -285> 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -284> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -283> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -282> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -281> 2020-01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -280> 2020-01-14T08:58:15.248+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -279> 2020-01-14T08:58:15.248+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -278> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started. -277> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20 -276> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0 -275> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096 -274> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -273> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -272> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -271> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0 -270> 2020-01-14T08:58:15.816+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -269> 2020-01-14T08:58:15.816+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -268> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -267> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -266> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -265> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process -264> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -263> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -262> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -261> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -260> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -259> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -258> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 20 -257> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -256> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -255> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1 -254> 2020-01-14T08:58:15.816+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -253> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -252> 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084 -251> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -250> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -249> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -248> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075 -247> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -246> 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084 -245> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -244> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -243> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -242> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88 -241> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -240> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -239> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -238> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -237> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 -236> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -235> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -234> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -233> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -232> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -231> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -230> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 -229> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -228> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -227> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -226> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -225> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -224> 2020-01-14T08:58:25.828+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -223> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -222> 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -221> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -220> 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -219> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -218> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -217> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -216> 2020-01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -215> 2020-01-14T08:58:26.636+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -214> 2020-01-14T08:58:26.636+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -213> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started. -212> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21 -211> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40 -210> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096 -209> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -208> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -207> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -206> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -205> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -204> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -203> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0 -202> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -201> 2020-01-14T08:58:26.640+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -200> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -199> 2020-01-14T08:58:26.640+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -198> 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084 -197> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -196> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -195> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -194> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075 -193> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process -192> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -191> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -190> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -189> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -188> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -187> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -186> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 21 -185> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -184> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -183> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1 -182> 2020-01-14T08:58:26.640+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -181> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -180> 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084 -179> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -178> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -177> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -176> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60 -175> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -174> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -173> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -172> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -171> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 -170> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -169> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -168> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -167> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -166> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -165> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -164> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 -163> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -162> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -161> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -160> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -159> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -158> 2020-01-14T08:58:36.652+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -157> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -156> 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -155> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -154> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -153> 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -152> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -151> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -150> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -149> 2020-01-14T08:58:36.652+0800 ffff672f1e80 1 -- v2:172.19.36.252:4567/0 reap_dead start -148> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880 -147> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00 -146> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180 -145> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600 -144> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80 -143> 2020-01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -142> 2020-01-14T08:58:38.260+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -141> 2020-01-14T08:58:38.260+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -140> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started. -139> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22 -138> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0 -137> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096 -136> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -135> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -134> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -133> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0 -132> 2020-01-14T08:58:38.264+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -131> 2020-01-14T08:58:38.264+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -130> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -129> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -128> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -127> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process -126> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -125> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -124> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -123> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -122> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -121> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -120> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 22 -119> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -118> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -117> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1 -116> 2020-01-14T08:58:38.264+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -115> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -114> 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084 -113> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -112> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -111> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -110> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075 -109> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -108> 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084 -107> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -106> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -105> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -104> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38 -103> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -102> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -101> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -100> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -99> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 -98> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -97> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -96> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -95> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -94> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -93> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -92> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 -91> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -90> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -89> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -88> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -87> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -86> 2020-01-14T08:58:48.276+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -85> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -84> 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -83> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -82> 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -81> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -80> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -79> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -78> 2020-01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -77> 2020-01-14T08:58:51.484+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -76> 2020-01-14T08:58:51.484+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -75> 2020-01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started. -74> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23 -73> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0 -72> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096 -71> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -70> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -69> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -68> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -67> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -66> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0 -65> 2020-01-14T08:58:51.492+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -64> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -63> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -62> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -61> 2020-01-14T08:58:51.492+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -60> 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084 -59> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -58> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -57> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -56> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075 -55> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process -54> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -53> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -52> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -51> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -50> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -49> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -48> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 23 -47> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -46> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -45> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1 -44> 2020-01-14T08:58:51.492+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -43> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -42> 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084 -41> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -40> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -39> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -38> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10 -37> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -36> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -35> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -34> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -33> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 -32> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -31> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -30> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -29> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -28> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -27> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -26> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 -25> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -24> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -23> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -22> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -21> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -20> 2020-01-14T08:59:01.504+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -19> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -18> 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -17> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -16> 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -15> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -14> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -13> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -12> 2020-01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -11> 2020-01-14T08:59:09.088+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -10> 2020-01-14T08:59:09.088+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -9> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started. -8> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24 -7> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280 -6> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768 -5> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768 -4> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers. -3> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR -2> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0 -1> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280 0> 2020-01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) ** in thread ffff672f1e80 thread_name:msgr-worker-0 ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev) 1: (__kernel_rt_sigreturn()+0) [0xffffa8c315c0] 2: (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr&, std::shared_ptr&, RDMAWorker*)+0x18c) [0xaaaac62ec874] 3: (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc] 4: (Processor::accept()+0x11c) [0xaaaac60396b4] 5: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x51c) [0xaaaac604167c] 6: (()+0x469a20) [0xaaaac6047a20] 7: (()+0xc9ed4) [0xffffa8719ed4] 8: (()+0x7088) [0xffffa8bcd088] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 20/20 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events --- --- begin dump of recent events --- -588> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command assert hook 0xaaab0c49e700 -587> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command abort hook 0xaaab0c49e700 -586> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perfcounters_dump hook 0xaaab0c49e700 -585> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command 1 hook 0xaaab0c49e700 -584> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf dump hook 0xaaab0c49e700 -583> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perfcounters_schema hook 0xaaab0c49e700 -582> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf histogram dump hook 0xaaab0c49e700 -581> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command 2 hook 0xaaab0c49e700 -580> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf schema hook 0xaaab0c49e700 -579> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf histogram schema hook 0xaaab0c49e700 -578> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command perf reset hook 0xaaab0c49e700 -577> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config show hook 0xaaab0c49e700 -576> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config help hook 0xaaab0c49e700 -575> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config set hook 0xaaab0c49e700 -574> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config unset hook 0xaaab0c49e700 -573> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config get hook 0xaaab0c49e700 -572> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config diff hook 0xaaab0c49e700 -571> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command config diff get hook 0xaaab0c49e700 -570> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command injectargs hook 0xaaab0c49e700 -569> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log flush hook 0xaaab0c49e700 -568> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log dump hook 0xaaab0c49e700 -567> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command log reopen hook 0xaaab0c49e700 -566> 2020-01-14T08:57:33.240+0800 ffffa8c24010 5 asok(0xaaab0c51e000) register_command dump_mempools hook 0xaaab0d110068 -565> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -564> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -563> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding auth protocol: cephx -562> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -561> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -560> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -559> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -558> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -557> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -556> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -555> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -554> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -553> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -552> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: crc -551> 2020-01-14T08:57:33.252+0800 ffffa8c24010 5 AuthRegistry(0xaaab0c592148) adding con mode: secure -550> 2020-01-14T08:57:33.252+0800 ffffa8c24010 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -549> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband verify_prereq ms_async_rdma_enable_hugepage value is: 0 -548> 2020-01-14T08:57:33.252+0800 ffffa8c24010 20 Infiniband Infiniband constructing Infiniband... -547> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack RDMAStack constructing RDMAStack... -546> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 RDMAStack creating RDMAStack:0xaaab0c4debc0 with dispatcher:0xaaab0d131df0 -545> 2020-01-14T08:57:33.272+0800 ffff672f1e80 2 Event(0xaaab0d110608 nevent=5000 time_id=1).set_owner center_id=0 owner=281472412884608 -544> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=5 mask=1 original mask is 0 -543> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=5 cur_mask=0 add_mask=1 to 4 -542> 2020-01-14T08:57:33.272+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=5 mask=1 original mask is 1 -541> 2020-01-14T08:57:33.272+0800 ffff672f1e80 10 stack operator() starting -540> 2020-01-14T08:57:33.272+0800 ffff66af0e80 2 Event(0xaaab0d1108c8 nevent=5000 time_id=1).set_owner center_id=1 owner=281472404491904 -539> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=8 mask=1 original mask is 0 -538> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=8 cur_mask=0 add_mask=1 to 7 -537> 2020-01-14T08:57:33.272+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=8 mask=1 original mask is 1 -536> 2020-01-14T08:57:33.272+0800 ffff66af0e80 10 stack operator() starting -535> 2020-01-14T08:57:33.272+0800 ffff662efe80 2 Event(0xaaab0d110b88 nevent=5000 time_id=1).set_owner center_id=2 owner=281472396099200 -534> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0 -533> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10 -532> 2020-01-14T08:57:33.272+0800 ffff662efe80 20 Event(0xaaab0d110b88 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1 -531> 2020-01-14T08:57:33.272+0800 ffff662efe80 10 stack operator() starting -530> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -529> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -528> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding auth protocol: cephx -527> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -526> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -525> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -524> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -523> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -522> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -521> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -520> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -519> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -518> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -517> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: crc -516> 2020-01-14T08:57:33.272+0800 ffffa8c24010 5 AuthRegistry(0xfffffeb401b8) adding con mode: secure -515> 2020-01-14T08:57:33.272+0800 ffffa8c24010 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -514> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bind v2:172.19.36.252:4567/0 -513> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 -512> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv Network Stack is not ready for bind yet - postponed -511> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- ready -510> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 -- bindv v2:172.19.36.252:4567/0 -509> 2020-01-14T08:57:33.272+0800 ffffa8c24010 10 Processor -- bind v2:172.19.36.252:4567/0 -508> 2020-01-14T08:57:33.272+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -507> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband binding_port found active port 1 -506> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 4096 receive buffers -505> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init assigning: 1024 send buffers -504> 2020-01-14T08:57:33.280+0800 ffff672f1e80 1 Infiniband init device allow 4194304 completion entries -503> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. -502> 2020-01-14T08:57:33.320+0800 ffff672f1e80 20 Infiniband init started. -501> 2020-01-14T08:57:33.324+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4da9c0 -500> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Infiniband init successfully create cq=0xaaab0c4daa80 -499> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 RDMAStack polling going to poll tx cq: 0xaaab0d1e1b30 rx cq: 0xaaab0d1e1b60 -498> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 RDMAServerSocketImpl listen bind to 172.19.36.252:4567 on port 4567 -497> 2020-01-14T08:57:33.328+0800 ffffa8c24010 10 Processor -- bind bound to v2:172.19.36.252:4567/0 -496> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 learned_addr learned my addr v2:172.19.36.252:4567/0 (peer_addr_for_me v2:172.19.36.252:4567/0) -495> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 _finish_bind bind my_addrs is v2:172.19.36.252:4567/0 -494> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 Processor -- start -493> 2020-01-14T08:57:33.328+0800 ffffa8c24010 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -492> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event started fd=27 mask=1 original mask is 0 -491> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 EpollDriver.add_event add event fd=27 cur_mask=0 add_mask=1 to 4 -490> 2020-01-14T08:57:33.328+0800 ffff672f1e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).create_file_event create event end fd=27 mask=1 original mask is 1 -489> 2020-01-14T08:57:33.328+0800 ffffa8c24010 1 -- v2:172.19.36.252:4567/0 start start -488> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. -487> 2020-01-14T08:57:33.328+0800 ffff65780e80 20 Infiniband rearm_notify started. -486> 2020-01-14T08:57:34.836+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -485> 2020-01-14T08:57:34.836+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -484> 2020-01-14T08:57:34.836+0800 ffff672f1e80 20 Infiniband init started. -483> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 17 -482> 2020-01-14T08:57:34.840+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52ea00 -481> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 17 post SQ WR 4096 -480> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -479> 2020-01-14T08:57:35.132+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -478> 2020-01-14T08:57:35.132+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -477> 2020-01-14T08:57:35.136+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55558/0 -476> 2020-01-14T08:57:35.136+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -475> 2020-01-14T08:57:35.136+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -474> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -473> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -472> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -471> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_ACCEPTING l=0).process -470> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -469> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -468> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -467> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -466> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -465> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -464> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 17 -463> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -462> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -461> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 17 r = -1 -460> 2020-01-14T08:57:35.136+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -459> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -458> 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 0, fe8000000000000002182dfffe000084 -457> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -456> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -455> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -454> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 17, 0, 64, fe8000000000000002182dfffe000075 -453> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -452> 2020-01-14T08:57:35.136+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 64, 11581620, 17, fe8000000000000002182dfffe000084 -451> 2020-01-14T08:57:35.136+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -450> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -449> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -448> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 17 0xaaab0d1fbfd8 -447> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -446> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -445> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -444> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -443> 2020-01-14T08:57:35.136+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 -442> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 17 tcp_fd: 28 notify_fd: 29 -441> 2020-01-14T08:57:35.592+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -440> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -439> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -438> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -437> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -436> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 17 r = 0 -435> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -434> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -433> 2020-01-14T08:57:35.592+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -432> 2020-01-14T08:57:35.592+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -431> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -430> 2020-01-14T08:57:35.596+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -429> 2020-01-14T08:57:35.596+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -428> 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -427> 2020-01-14T08:57:35.596+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -426> 2020-01-14T08:57:35.596+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -425> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -424> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -423> 2020-01-14T08:57:35.596+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -422> 2020-01-14T08:57:35.596+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -421> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling got tx cq event. -420> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack polling tx completion queue got 1 responses. -419> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 RDMAStack handle_tx_event QP number: 17 len: 0 status: RETRY_EXC_ERR -418> 2020-01-14T08:57:45.048+0800 ffff65780e80 1 RDMAStack handle_tx_event Responder ACK timeout, possible disconnect, or Remote QP in bad state WCE status(12): RETRY_EXC_ERR WCE QP number 17 Opcode 0 wr_id: 0xaaab0d1fbfd8 -417> 2020-01-14T08:57:45.048+0800 ffff65780e80 10 RDMAStack polling finally delete qp = 0xaaab0c502800 -416> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 17 left SQ WR 4096 -415> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52ea00 -414> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. -413> 2020-01-14T08:57:45.048+0800 ffff65780e80 20 Infiniband rearm_notify started. -412> 2020-01-14T08:57:54.004+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -411> 2020-01-14T08:57:54.004+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -410> 2020-01-14T08:57:54.004+0800 ffff672f1e80 20 Infiniband init started. -409> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 18 -408> 2020-01-14T08:57:54.008+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f900 -407> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 18 post SQ WR 4096 -406> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -405> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -404> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -403> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -402> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -401> 2020-01-14T08:57:54.012+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55566/0 -400> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -399> 2020-01-14T08:57:54.012+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -398> 2020-01-14T08:57:54.012+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -397> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -396> 2020-01-14T08:57:54.012+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -395> 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 0, fe8000000000000002182dfffe000084 -394> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -393> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -392> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -391> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 18, 2116118, 65, fe8000000000000002182dfffe000075 -390> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_ACCEPTING l=0).process -389> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -388> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -387> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -386> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -385> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -384> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -383> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 18 -382> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -381> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -380> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 18 r = -1 -379> 2020-01-14T08:57:54.012+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -378> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -377> 2020-01-14T08:57:54.012+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 65, 0, 18, fe8000000000000002182dfffe000084 -376> 2020-01-14T08:57:54.012+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -375> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -374> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -373> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 18 0xaaab0d1fbfd8 -372> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -371> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -370> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -369> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -368> 2020-01-14T08:57:54.012+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 -367> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 18 tcp_fd: 28 notify_fd: 29 -366> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -365> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -364> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -363> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -362> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -361> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 18 r = 0 -360> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -359> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d125600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -358> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -357> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -356> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -355> 2020-01-14T08:58:04.024+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -354> 2020-01-14T08:58:04.024+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -353> 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -352> 2020-01-14T08:58:04.024+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -351> 2020-01-14T08:58:04.024+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d125600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -350> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -349> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -348> 2020-01-14T08:58:04.024+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -347> 2020-01-14T08:58:04.024+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -346> 2020-01-14T08:58:04.228+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -345> 2020-01-14T08:58:04.228+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -344> 2020-01-14T08:58:04.228+0800 ffff672f1e80 20 Infiniband init started. -343> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 19 -342> 2020-01-14T08:58:04.232+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52f040 -341> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 19 post SQ WR 4096 -340> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -339> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -338> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -337> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -336> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -335> 2020-01-14T08:58:04.236+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55576/0 -334> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -333> 2020-01-14T08:58:04.236+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -332> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -331> 2020-01-14T08:58:04.236+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -330> 2020-01-14T08:58:04.236+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -329> 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 0, fe8000000000000002182dfffe000084 -328> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -327> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -326> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -325> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 19, 5515815, 66, fe8000000000000002182dfffe000075 -324> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_ACCEPTING l=0).process -323> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -322> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -321> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -320> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -319> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -318> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -317> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 19 -316> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -315> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -314> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 19 r = -1 -313> 2020-01-14T08:58:04.236+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -312> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -311> 2020-01-14T08:58:04.236+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 66, 2116118, 19, fe8000000000000002182dfffe000084 -310> 2020-01-14T08:58:04.236+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -309> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -308> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -307> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 19 0xaaab0d1fbfb0 -306> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -305> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -304> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -303> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -302> 2020-01-14T08:58:04.236+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 -301> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 19 tcp_fd: 28 notify_fd: 29 -300> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -299> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -298> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -297> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -296> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -295> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 19 r = 0 -294> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -293> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 msgr2=0xaaab0d125b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -292> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -291> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -290> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -289> 2020-01-14T08:58:14.248+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -288> 2020-01-14T08:58:14.248+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -287> 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -286> 2020-01-14T08:58:14.248+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -285> 2020-01-14T08:58:14.248+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf180 0xaaab0d125b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -284> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -283> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -282> 2020-01-14T08:58:14.248+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -281> 2020-01-14T08:58:14.248+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -280> 2020-01-14T08:58:15.248+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -279> 2020-01-14T08:58:15.248+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -278> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init started. -277> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 20 -276> 2020-01-14T08:58:15.248+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52edc0 -275> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 20 post SQ WR 4096 -274> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -273> 2020-01-14T08:58:15.816+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -272> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -271> 2020-01-14T08:58:15.816+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55580/0 -270> 2020-01-14T08:58:15.816+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -269> 2020-01-14T08:58:15.816+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -268> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -267> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -266> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -265> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process -264> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -263> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -262> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -261> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -260> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -259> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -258> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 20 -257> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -256> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -255> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 20 r = -1 -254> 2020-01-14T08:58:15.816+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -253> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -252> 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 0, fe8000000000000002182dfffe000084 -251> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -250> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -249> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -248> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 20, 10238434, 67, fe8000000000000002182dfffe000075 -247> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -246> 2020-01-14T08:58:15.816+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 67, 5515815, 20, fe8000000000000002182dfffe000084 -245> 2020-01-14T08:58:15.816+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -244> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -243> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -242> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 20 0xaaab0d1fbf88 -241> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -240> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -239> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -238> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -237> 2020-01-14T08:58:15.816+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 -236> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 20 tcp_fd: 28 notify_fd: 29 -235> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -234> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -233> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -232> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -231> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -230> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 20 r = 0 -229> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -228> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -227> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -226> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -225> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -224> 2020-01-14T08:58:25.828+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -223> 2020-01-14T08:58:25.828+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -222> 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -221> 2020-01-14T08:58:25.828+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -220> 2020-01-14T08:58:25.828+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cf600 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -219> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -218> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -217> 2020-01-14T08:58:25.828+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -216> 2020-01-14T08:58:25.828+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -215> 2020-01-14T08:58:26.636+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -214> 2020-01-14T08:58:26.636+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -213> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init started. -212> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 21 -211> 2020-01-14T08:58:26.636+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52eb40 -210> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 21 post SQ WR 4096 -209> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -208> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -207> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -206> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -205> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -204> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -203> 2020-01-14T08:58:26.640+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55586/0 -202> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -201> 2020-01-14T08:58:26.640+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -200> 2020-01-14T08:58:26.640+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -199> 2020-01-14T08:58:26.640+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -198> 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 0, fe8000000000000002182dfffe000084 -197> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -196> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -195> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -194> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 21, 11225430, 68, fe8000000000000002182dfffe000075 -193> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_ACCEPTING l=0).process -192> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -191> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -190> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -189> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -188> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -187> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -186> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 21 -185> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -184> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -183> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 21 r = -1 -182> 2020-01-14T08:58:26.640+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -181> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -180> 2020-01-14T08:58:26.640+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 68, 10238434, 21, fe8000000000000002182dfffe000084 -179> 2020-01-14T08:58:26.640+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -178> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -177> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -176> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 21 0xaaab0d1fbf60 -175> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -174> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -173> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -172> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -171> 2020-01-14T08:58:26.640+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 -170> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 21 tcp_fd: 28 notify_fd: 29 -169> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -168> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -167> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -166> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -165> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -164> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 21 r = 0 -163> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -162> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 msgr2=0xaaab0d126680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -161> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -160> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -159> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -158> 2020-01-14T08:58:36.652+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -157> 2020-01-14T08:58:36.652+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -156> 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -155> 2020-01-14T08:58:36.652+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -154> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 Event(0xaaab0d110608 nevent=5000 time_id=1).wakeup -153> 2020-01-14T08:58:36.652+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4cfa80 0xaaab0d126680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -152> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -151> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -150> 2020-01-14T08:58:36.652+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -149> 2020-01-14T08:58:36.652+0800 ffff672f1e80 1 -- v2:172.19.36.252:4567/0 reap_dead start -148> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ce880 -147> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4ced00 -146> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf180 -145> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cf600 -144> 2020-01-14T08:58:36.652+0800 ffff672f1e80 5 -- v2:172.19.36.252:4567/0 reap_dead delete 0xaaab0c4cfa80 -143> 2020-01-14T08:58:36.652+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -142> 2020-01-14T08:58:38.260+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -141> 2020-01-14T08:58:38.260+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -140> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init started. -139> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 22 -138> 2020-01-14T08:58:38.260+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e8c0 -137> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 22 post SQ WR 4096 -136> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -135> 2020-01-14T08:58:38.264+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -134> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -133> 2020-01-14T08:58:38.264+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55594/0 -132> 2020-01-14T08:58:38.264+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -131> 2020-01-14T08:58:38.264+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -130> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -129> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -128> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -127> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_ACCEPTING l=0).process -126> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -125> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -124> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -123> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -122> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -121> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -120> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 22 -119> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -118> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -117> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 22 r = -1 -116> 2020-01-14T08:58:38.264+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not active. len: 4096 -115> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -114> 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 0, fe8000000000000002182dfffe000084 -113> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -112> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -111> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -110> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 22, 11581620, 69, fe8000000000000002182dfffe000075 -109> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -108> 2020-01-14T08:58:38.264+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 69, 11225430, 22, fe8000000000000002182dfffe000084 -107> 2020-01-14T08:58:38.264+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -106> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -105> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -104> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 22 0xaaab0d1fbf38 -103> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -102> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -101> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -100> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -99> 2020-01-14T08:58:38.264+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 -98> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 22 tcp_fd: 28 notify_fd: 29 -97> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -96> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -95> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -94> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -93> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -92> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 22 r = 0 -91> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -90> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 msgr2=0xaaab0d126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -89> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -88> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -87> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -86> 2020-01-14T08:58:48.276+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -85> 2020-01-14T08:58:48.276+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -84> 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -83> 2020-01-14T08:58:48.276+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -82> 2020-01-14T08:58:48.276+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ce880 0xaaab0d126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -81> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -80> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -79> 2020-01-14T08:58:48.276+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -78> 2020-01-14T08:58:48.276+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -77> 2020-01-14T08:58:51.484+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -76> 2020-01-14T08:58:51.484+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -75> 2020-01-14T08:58:51.484+0800 ffff672f1e80 20 Infiniband init started. -74> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 23 -73> 2020-01-14T08:58:51.488+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e3c0 -72> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Infiniband init initialize no SRQ Queue Pair, qp number: 23 post SQ WR 4096 -71> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -70> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 28 -69> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 Processor -- accept accepted incoming on sd 29 -68> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0 -67> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 7 -66> 2020-01-14T08:58:51.492+0800 ffff672f1e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_NONE l=0).accept sd=29 listen_addr v2:172.19.36.252:4567/0 peer_addr v2:172.19.36.251:55602/0 -65> 2020-01-14T08:58:51.492+0800 ffff672f1e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept -64> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1 -63> 2020-01-14T08:58:51.492+0800 ffff672f1e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).wakeup -62> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -61> 2020-01-14T08:58:51.492+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -60> 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 0, fe8000000000000002182dfffe000084 -59> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr Choosing gid_index 0, sl 3 -58> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rtr transition to RTR state successfully. -57> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Infiniband modify_qp_to_rts transition to RTS state successfully. -56> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 Infiniband send_cm_meta sending: 0, 23, 13658313, 70, fe8000000000000002182dfffe000075 -55> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_ACCEPTING l=0).process -54> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event started fd=29 mask=1 original mask is 0 -53> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 EpollDriver.add_event add event fd=29 cur_mask=0 add_mask=1 to 7 -52> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 Event(0xaaab0d1108c8 nevent=5000 time_id=1).create_file_event create event end fd=29 mask=1 original mask is 1 -51> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).read_event -50> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=START_ACCEPT pgs=0 cs=0 l=0 rx=0 tx=0).start_server_banner_exchange -49> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._banner_exchange -48> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl send fake send to upper, QP: 23 -47> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 -46> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 -45> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 0 in 23 r = -1 -44> 2020-01-14T08:58:51.492+0800 ffff66af0e80 1 RDMAConnectedSocketImpl read when ib not connected. len: 4096 -43> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -42> 2020-01-14T08:58:51.492+0800 ffff66af0e80 5 Infiniband recv_cm_meta recevd: 0, 70, 11581620, 23, fe8000000000000002182dfffe000084 -41> 2020-01-14T08:58:51.492+0800 ffff66af0e80 10 RDMAConnectedSocketImpl handle_connection handshake of rdma is done. server connected: 1 -40> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit we need 26 bytes. iov size: 2 -39> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1 -38> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request QP: 23 0xaaab0d1fbf10 -37> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS -36> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl submit finished sending 26 bytes. -35> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -34> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -33> 2020-01-14T08:58:51.492+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 -32> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl handle_connection QP: 23 tcp_fd: 28 notify_fd: 29 -31> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 Infiniband recv_cm_meta got disconnect message -30> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl handle_connection recv handshake msg failed. -29> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 RDMAConnectedSocketImpl fault tcp fd 28 -28> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process -27> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=10 -26> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl read notify_fd : 1 in 23 r = 0 -25> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=29 : Unknown error -104 -24> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 -- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 msgr2=0xaaab0d126100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed -23> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner r=-1 -22> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) -21> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault -20> 2020-01-14T08:59:01.504+0800 ffff66af0e80 2 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._fault with nothing to send and in the half accept state just closed -19> 2020-01-14T08:59:01.504+0800 ffff66af0e80 1 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).stop -18> 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state -17> 2020-01-14T08:59:01.504+0800 ffff66af0e80 10 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).discard_out_queue started -16> 2020-01-14T08:59:01.504+0800 ffff66af0e80 5 --2- v2:172.19.36.252:4567/0 >> conn(0xaaab0c4ced00 0xaaab0d126100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).reset_recv_state reseting crypto handlers -15> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=29 cur_mask=1 delmask=3 to 7 -14> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct. -13> 2020-01-14T08:59:01.504+0800 ffff66af0e80 20 EpollDriver.del_event del event fd=28 cur_mask=1 delmask=1 to 7 -12> 2020-01-14T08:59:01.504+0800 ffff66af0e80 -1 Infiniband modify_qp_to_error failed to transition to ERROR state: (22) Invalid argument -11> 2020-01-14T08:59:09.088+0800 ffff672f1e80 10 Processor -- accept listen_fd=27 -10> 2020-01-14T08:59:09.088+0800 ffff672f1e80 15 RDMAServerSocketImpl accept -9> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init started. -8> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband modify_qp_to_init successfully switch to INIT state Queue Pair, qp number: 24 -7> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband init successfully create queue pair: qp=0xaaab0c52e280 -6> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 32768 limit: 32768 -5> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband can_alloc WARNING: OUT OF RX BUFFERS: allocated: 24576 requested: 16384 limit: 32768 -4> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband post_chunks_to_rq WARNING: out of memory. Request 4096 rx buffers. Only get 0 rx buffers. -3> 2020-01-14T08:59:09.088+0800 ffff672f1e80 -1 Infiniband init intialize no SRQ Queue Pair, qp number: 24 fatal error: can't post SQ WR -2> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy Queue Pair, qp number: 24 left SQ WR 0 -1> 2020-01-14T08:59:09.088+0800 ffff672f1e80 20 Infiniband ~QueuePair destroy qp=0xaaab0c52e280 0> 2020-01-14T08:59:09.100+0800 ffff672f1e80 -1 *** Caught signal (Segmentation fault) ** in thread ffff672f1e80 thread_name:msgr-worker-0 ceph version 15.0.0-8506-g0277d9184e (0277d9184ee3f681fad7812b4275e8d97353353d) octopus (dev) 1: (__kernel_rt_sigreturn()+0) [0xffffa8c315c0] 2: (RDMAConnectedSocketImpl::RDMAConnectedSocketImpl(CephContext*, std::shared_ptr&, std::shared_ptr&, RDMAWorker*)+0x18c) [0xaaaac62ec874] 3: (RDMAServerSocketImpl::accept(ConnectedSocket*, SocketOptions const&, entity_addr_t*, Worker*)+0x124) [0xaaaac62f4dcc] 4: (Processor::accept()+0x11c) [0xaaaac60396b4] 5: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x51c) [0xaaaac604167c] 6: (()+0x469a20) [0xaaaac6047a20] 7: (()+0xc9ed4) [0xffffa8719ed4] 8: (()+0x7088) [0xffffa8bcd088] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 20/20 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file /var/lib/ceph/crash/2020-01-14T00:59:09.101446Z_5af684cd-6ecb-4f45-8495-307fc79fc807/log --- end dump of recent events --- Segmentation fault (core dumped) root@node2:~/testceph# root@node2:~/testceph# root@node2:~/testceph# root@node2:~/testceph#