ceph-client.smp-016.log - Ceph - Ceph

Bug #22944 » ceph-client.smp-016.log

client log - Radosław Piliszek, 02/13/2018 08:01 AM

 -02-13 08:52:17.174866 7f9971b2d700  1 RDMAStack RDMAStack ms_async_rdma_enable_hugepage value is: 0
 -02-13 08:52:17.174877 7f9971b2d700 20 RDMAStack RDMAStack constructing RDMAStack...
 -02-13 08:52:17.174898 7f9971b2d700 20 RDMAStack  creating RDMAStack:0x7f996c0b54a8 with dispatcher:0x7f996c13e350
 -02-13 08:52:17.175036 7f996b7fe700  2 Event(0x7f996c0b55a0 nevent=5000 time_id=1).set_owner idx=0 owner=140296910268160
 -02-13 08:52:17.175039 7f996affd700  2 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).set_owner idx=1 owner=140296901875456
 -02-13 08:52:17.175076 7f996b7fe700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).create_file_event create event started fd=4 mask=1 original mask is 0
 -02-13 08:52:17.175073 7f99627fc700  2 Event(0x7f996c110840 nevent=5000 time_id=1).set_owner idx=2 owner=140296759265024
 -02-13 08:52:17.175079 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=7 mask=1 original mask is 0
 -02-13 08:52:17.175081 7f996b7fe700 20 EpollDriver.add_event add event fd=4 cur_mask=0 add_mask=1 to 3
 -02-13 08:52:17.175084 7f996affd700 20 EpollDriver.add_event add event fd=7 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:17.175083 7f99627fc700 20 Event(0x7f996c110840 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0
 -02-13 08:52:17.175087 7f996b7fe700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).create_file_event create event end fd=4 mask=1 original mask is 1
 -02-13 08:52:17.175088 7f99627fc700 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10
 -02-13 08:52:17.175090 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=7 mask=1 original mask is 1
 -02-13 08:52:17.175093 7f996affd700 10 stack operator() starting
 -02-13 08:52:17.175092 7f99627fc700 20 Event(0x7f996c110840 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1
 -02-13 08:52:17.175093 7f996b7fe700 10 stack operator() starting
 -02-13 08:52:17.175096 7f99627fc700 10 stack operator() starting
 -02-13 08:52:17.175323 7f9971b2d700 10 -- - ready -
 -02-13 08:52:17.175330 7f9971b2d700  1  Processor -- start
 -02-13 08:52:17.175385 7f9971b2d700  1 -- - start start
 -02-13 08:52:17.176141 7f9971b2d700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
 -02-13 08:52:17.176170 7f9971b2d700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
 -02-13 08:52:17.176180 7f9971b2d700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:17.176191 7f9971b2d700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f996c154ec0
 -02-13 08:52:17.176207 7f9971b2d700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
 -02-13 08:52:17.176212 7f9971b2d700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:17.176208 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:17.176227 7f9971b2d700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f996c152fc0 con 0
 -02-13 08:52:17.176241 7f9971b2d700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f996c152fc0
 -02-13 08:52:17.177693 7f996affd700  1 Infiniband binding_port found active port 1
 -02-13 08:52:17.177728 7f996affd700  1 Infiniband init assigning: 1024 receive buffers
 -02-13 08:52:17.177731 7f996affd700  1 Infiniband init assigning: 1024 send buffers
 -02-13 08:52:17.177733 7f996affd700  1 Infiniband init device allow 4194303 completion entries
 -02-13 08:52:17.250939 7f996affd700 20 Infiniband init started.
 -02-13 08:52:17.250969 7f996affd700 20 Infiniband init started.
 -02-13 08:52:17.252142 7f996affd700 20 Infiniband init successfully create cq=0x7f99540082b0
 -02-13 08:52:17.253299 7f996affd700 20 Infiniband init successfully create cq=0x7f99540086a0
 -02-13 08:52:17.253399 7f996affd700 20 Infiniband init started.
 -02-13 08:52:17.253487 7f99637fe700 20 RDMAStack polling going to poll tx cq: 0x7f9954008280 rx cq: 0x7f9954008670
 -02-13 08:52:17.253506 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:17.253507 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:17.253528 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f9954008f30
 -02-13 08:52:17.253985 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f9954008f30
 -02-13 08:52:17.254020 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
 -02-13 08:52:17.255550 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
 -02-13 08:52:17.255564 7f996affd700 10 Infiniband send_msg sending: 79, 262717, 0, 0, fe80000000000000f45214030035f771
 -02-13 08:52:17.255604 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=18 mask=1 original mask is 0
 -02-13 08:52:17.255608 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:17.255613 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=18 mask=1 original mask is 1
 -02-13 08:52:17.255618 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=17 mask=1 original mask is 0
 -02-13 08:52:17.255619 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:17.255621 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=17 mask=1 original mask is 1
 -02-13 08:52:17.255624 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:17.255642 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
 -02-13 08:52:17.255669 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:17.255675 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:17.256391 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262717 tcp_fd: 18 notify_fd: 17
 -02-13 08:52:17.256403 7f996affd700  5 Infiniband recv_msg recevd: 4, 0, 2790260736, 2140754944, ???j
 -02-13 08:52:17.256406 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 0, 2790260736, 4, 2140754944>
 -02-13 08:52:17.256410 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
 -02-13 08:52:17.256484 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
 -02-13 08:52:17.256547 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
 -02-13 08:52:17.256551 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954008dd0 with qp:0x7f9954008f30
 -02-13 08:52:17.256553 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262717
 -02-13 08:52:17.256557 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
 -02-13 08:52:17.256563 7f996affd700 10 Infiniband send_msg sending: 79, 262717, 0, 0, fe80000000000000f45214030035f771
 -02-13 08:52:17.256585 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:17.256614 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
 -02-13 08:52:17.256619 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
 -02-13 08:52:17.256640 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262717
 -02-13 08:52:17.256644 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
 -02-13 08:52:17.256650 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -02-13 08:52:17.256653 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262717 0x7f9954019700
 -02-13 08:52:17.256674 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -02-13 08:52:17.256731 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
 -02-13 08:52:17.256733 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
 -02-13 08:52:17.256739 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=17 mask=2 original mask is 1
 -02-13 08:52:17.256741 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
 -02-13 08:52:17.256743 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=17 mask=2 original mask is 3
 -02-13 08:52:17.256747 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
 -02-13 08:52:17.256753 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:17.256761 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262717 r = 8
 -02-13 08:52:17.256766 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:17.256771 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
 -02-13 08:52:21.498227 7f99637fe700 20 RDMAStack polling got tx cq event.
 -02-13 08:52:21.498238 7f99637fe700 20 RDMAStack polling tx completion queue got 1 responses.
 -02-13 08:52:21.498245 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
 -02-13 08:52:21.498250 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
 -02-13 08:52:21.498265 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:21.498266 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:21.498306 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY
 -02-13 08:52:21.498340 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262717 r = 8
 -02-13 08:52:21.498345 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_bulk reading from fd=17 : Unknown error -104
 -02-13 08:52:21.498368 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_until read failed
 -02-13 08:52:21.498375 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection read banner and identify addresses failed
 -02-13 08:52:21.498391 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
 -02-13 08:52:21.498400 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -02-13 08:52:21.498404 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
 -02-13 08:52:21.498548 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).fault waiting 0.200000
 -02-13 08:52:21.698850 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:21.698888 7f996affd700 20 Infiniband init started.
 -02-13 08:52:21.699048 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f995402b330
 -02-13 08:52:21.699734 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f995402b330
 -02-13 08:52:21.699755 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
 -02-13 08:52:21.699925 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
 -02-13 08:52:21.699943 7f996affd700 10 Infiniband send_msg sending: 79, 262719, 2116118, 0, fe80000000000000f45214030035f771
 -02-13 08:52:21.699992 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=18 mask=1 original mask is 0
 -02-13 08:52:21.700003 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:21.700011 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=18 mask=1 original mask is 1
 -02-13 08:52:21.700015 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=17 mask=1 original mask is 0
 -02-13 08:52:21.700017 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:21.700038 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=17 mask=1 original mask is 1
 -02-13 08:52:21.700043 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:21.700054 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
 -02-13 08:52:21.701232 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262719 tcp_fd: 18 notify_fd: 17
 -02-13 08:52:21.701270 7f996affd700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???j
 -02-13 08:52:21.701275 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 0, 0, 4, 0>
 -02-13 08:52:21.701277 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
 -02-13 08:52:21.701357 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
 -02-13 08:52:21.701472 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
 -02-13 08:52:21.701480 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954000f70 with qp:0x7f995402b330
 -02-13 08:52:21.701483 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262719
 -02-13 08:52:21.701493 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
 -02-13 08:52:21.701508 7f996affd700 10 Infiniband send_msg sending: 79, 262719, 2116118, 0, fe80000000000000f45214030035f771
 -02-13 08:52:21.701607 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:21.701628 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
 -02-13 08:52:21.701653 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
 -02-13 08:52:21.701680 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262719
 -02-13 08:52:21.701683 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
 -02-13 08:52:21.701690 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -02-13 08:52:21.701692 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262719 0x7f9954019700
 -02-13 08:52:21.701697 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -02-13 08:52:21.701770 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
 -02-13 08:52:21.701775 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
 -02-13 08:52:21.701785 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=17 mask=2 original mask is 1
 -02-13 08:52:21.701789 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
 -02-13 08:52:21.701793 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=17 mask=2 original mask is 3
 -02-13 08:52:21.701797 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
 -02-13 08:52:21.701808 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:21.701818 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262719 r = 8
 -02-13 08:52:21.701828 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:21.701835 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
 -02-13 08:52:25.826826 7f99637fe700 20 RDMAStack polling got tx cq event.
 -02-13 08:52:25.826835 7f99637fe700 20 RDMAStack polling tx completion queue got 1 responses.
 -02-13 08:52:25.826837 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
 -02-13 08:52:25.826839 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
 -02-13 08:52:25.826845 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954008dd0
 -02-13 08:52:25.826847 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f9954008f30
 -02-13 08:52:25.826908 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY
 -02-13 08:52:25.826939 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262719 r = 8
 -02-13 08:52:25.826944 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_bulk reading from fd=17 : Unknown error -104
 -02-13 08:52:25.826956 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_until read failed
 -02-13 08:52:25.826962 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection read banner and identify addresses failed
 -02-13 08:52:25.826969 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
 -02-13 08:52:25.826978 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -02-13 08:52:25.826980 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
 -02-13 08:52:25.827137 7f99637fe700 10 RDMAStack handle_async_event event associated qp=0x7f995402b330 evt: last WQE reached
 -02-13 08:52:25.827145 7f99637fe700  1 RDMAStack handle_async_event it's not forwardly stopped by us, reenable=0x7f995402b0c0
 -02-13 08:52:25.827147 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
 -02-13 08:52:25.827151 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:25.827153 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:25.827165 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954000f70
 -02-13 08:52:25.827168 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f995402b330
 -02-13 08:52:25.827235 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).fault waiting 0.400000
 -02-13 08:52:26.227711 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:26.227757 7f996affd700 20 Infiniband init started.
 -02-13 08:52:26.227918 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f995402b330
 -02-13 08:52:26.228564 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f995402b330
 -02-13 08:52:26.228592 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
 -02-13 08:52:26.228764 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
 -02-13 08:52:26.228783 7f996affd700 10 Infiniband send_msg sending: 79, 262721, 5515815, 0, fe80000000000000f45214030035f771
 -02-13 08:52:26.228830 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
 -02-13 08:52:26.228842 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:26.228849 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
 -02-13 08:52:26.228853 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
 -02-13 08:52:26.228856 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:26.228860 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
 -02-13 08:52:26.228863 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:26.228875 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
 -02-13 08:52:26.229831 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262721 tcp_fd: 18 notify_fd: 17
 -02-13 08:52:26.229869 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754944, 2258632704, 2140754944, ???j
 -02-13 08:52:26.229874 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754944, 2258632704, 4, 2140754944>
 -02-13 08:52:26.229876 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
 -02-13 08:52:26.229970 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
 -02-13 08:52:26.230021 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
 -02-13 08:52:26.230029 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954000f70 with qp:0x7f995402b330
 -02-13 08:52:26.230032 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262721
 -02-13 08:52:26.230034 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
 -02-13 08:52:26.230048 7f996affd700 10 Infiniband send_msg sending: 79, 262721, 5515815, 2140754944, fe80000000000000f45214030035f771
 -02-13 08:52:26.230109 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:26.230129 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
 -02-13 08:52:26.230146 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
 -02-13 08:52:26.230172 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262721
 -02-13 08:52:26.230180 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
 -02-13 08:52:26.230187 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -02-13 08:52:26.230190 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262721 0x7f9954019700
 -02-13 08:52:26.230194 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -02-13 08:52:26.230248 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
 -02-13 08:52:26.230252 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
 -02-13 08:52:26.230261 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
 -02-13 08:52:26.230265 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
 -02-13 08:52:26.230269 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
 -02-13 08:52:26.230272 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
 -02-13 08:52:26.230283 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:26.230293 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262721 r = 8
 -02-13 08:52:26.230303 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:26.230310 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
 -02-13 08:52:27.176292 7f9968ff9700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
 -02-13 08:52:27.176315 7f9968ff9700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
 -02-13 08:52:27.176329 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
 -02-13 08:52:27.176341 7f9968ff9700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f996c152fc0
 -02-13 08:52:27.176365 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:27.176413 7f9968ff9700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
 -02-13 08:52:27.176437 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
 -02-13 08:52:27.176488 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -02-13 08:52:27.176492 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
 -02-13 08:52:27.176513 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
 -02-13 08:52:27.176534 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:27.176544 7f9968ff9700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f99340013d0
 -02-13 08:52:27.176570 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
 -02-13 08:52:27.176595 7f9968ff9700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9934006040 con 0
 -02-13 08:52:27.176594 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:27.176612 7f9968ff9700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f9934006040
 -02-13 08:52:27.176620 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:27.176621 7f996affd700 20 Infiniband init started.
 -02-13 08:52:27.176788 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f99540251f0
 -02-13 08:52:27.177307 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f99540251f0
 -02-13 08:52:27.177327 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
 -02-13 08:52:27.177512 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
 -02-13 08:52:27.177531 7f996affd700 10 Infiniband send_msg sending: 79, 262723, 10238434, 0, fe80000000000000f45214030035f771
 -02-13 08:52:27.177578 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
 -02-13 08:52:27.177589 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:27.177596 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
 -02-13 08:52:27.177600 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
 -02-13 08:52:27.177602 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:27.177606 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
 -02-13 08:52:27.177610 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:27.177622 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
 -02-13 08:52:27.177630 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:27.177648 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:27.178651 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262723 tcp_fd: 18 notify_fd: 17
 -02-13 08:52:27.178675 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754944, 2258632704, 2140754944, ???j
 -02-13 08:52:27.178698 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754944, 2258632704, 4, 2140754944>
 -02-13 08:52:27.178702 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
 -02-13 08:52:27.178760 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
 -02-13 08:52:27.178835 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
 -02-13 08:52:27.178842 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f995402b1d0 with qp:0x7f99540251f0
 -02-13 08:52:27.178845 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262723
 -02-13 08:52:27.178850 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
 -02-13 08:52:27.178862 7f996affd700 10 Infiniband send_msg sending: 79, 262723, 10238434, 2140754944, fe80000000000000f45214030035f771
 -02-13 08:52:27.178908 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:27.178925 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
 -02-13 08:52:27.178932 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
 -02-13 08:52:27.178959 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262723
 -02-13 08:52:27.178962 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
 -02-13 08:52:27.178968 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -02-13 08:52:27.178972 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262723 0x7f99540196e0
 -02-13 08:52:27.178977 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -02-13 08:52:27.179045 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
 -02-13 08:52:27.179051 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
 -02-13 08:52:27.179066 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
 -02-13 08:52:27.179071 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
 -02-13 08:52:27.179075 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
 -02-13 08:52:27.179078 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
 -02-13 08:52:27.179089 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:27.179099 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262723 r = 8
 -02-13 08:52:27.179108 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:27.179116 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
 -02-13 08:52:30.176830 7f9968ff9700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
 -02-13 08:52:30.176843 7f9968ff9700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
 -02-13 08:52:30.176861 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
 -02-13 08:52:30.176867 7f9968ff9700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f9934006040
 -02-13 08:52:30.176881 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.176909 7f9968ff9700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
 -02-13 08:52:30.176938 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
 -02-13 08:52:30.176946 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.176943 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
 -02-13 08:52:30.176953 7f9968ff9700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f9934008210
 -02-13 08:52:30.176961 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -02-13 08:52:30.176971 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
 -02-13 08:52:30.176964 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
 -02-13 08:52:30.176988 7f9968ff9700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f993400bc10 con 0
 -02-13 08:52:30.177002 7f9968ff9700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f993400bc10
 -02-13 08:52:30.177067 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:30.177096 7f996affd700 20 Infiniband init started.
 -02-13 08:52:30.177273 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f9954038630
 -02-13 08:52:30.177885 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f9954038630
 -02-13 08:52:30.177906 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
 -02-13 08:52:30.178068 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
 -02-13 08:52:30.178086 7f996affd700 10 Infiniband send_msg sending: 79, 262725, 11225430, 0, fe80000000000000f45214030035f771
 -02-13 08:52:30.178132 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
 -02-13 08:52:30.178143 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:30.178150 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
 -02-13 08:52:30.178154 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
 -02-13 08:52:30.178157 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
 -02-13 08:52:30.178160 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
 -02-13 08:52:30.178178 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
 -02-13 08:52:30.178191 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
 -02-13 08:52:30.178199 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:30.178205 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:30.178996 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262725 tcp_fd: 18 notify_fd: 17
 -02-13 08:52:30.179018 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754947, 0, 0, ???j
 -02-13 08:52:30.179021 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754947, 0, 4, 0>
 -02-13 08:52:30.179022 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
 -02-13 08:52:30.179070 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
 -02-13 08:52:30.179127 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
 -02-13 08:52:30.179131 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954008dd0 with qp:0x7f9954038630
 -02-13 08:52:30.179133 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262725
 -02-13 08:52:30.179138 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
 -02-13 08:52:30.179147 7f996affd700 10 Infiniband send_msg sending: 79, 262725, 11225430, 2140754947, fe80000000000000f45214030035f771
 -02-13 08:52:30.179220 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:30.179239 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
 -02-13 08:52:30.179247 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
 -02-13 08:52:30.179286 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262725
 -02-13 08:52:30.179294 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
 -02-13 08:52:30.179301 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
 -02-13 08:52:30.179303 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262725 0x7f99540196c0
 -02-13 08:52:30.179308 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
 -02-13 08:52:30.179364 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
 -02-13 08:52:30.179368 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
 -02-13 08:52:30.179378 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
 -02-13 08:52:30.179381 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
 -02-13 08:52:30.179386 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
 -02-13 08:52:30.179389 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
 -02-13 08:52:30.179400 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
 -02-13 08:52:30.179410 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262725 r = 8
 -02-13 08:52:30.179419 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
 -02-13 08:52:30.179426 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
 -02-13 08:52:30.425890 7f99637fe700 20 RDMAStack polling got tx cq event.
 -02-13 08:52:30.425901 7f99637fe700 20 RDMAStack polling tx completion queue got 2 responses.
 -02-13 08:52:30.425903 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
 -02-13 08:52:30.425904 7f99637fe700  1 RDMAStack handle_tx_event missing qp_num=262721 discard event
 -02-13 08:52:30.425905 7f99637fe700  1 RDMAStack handle_tx_event Work Request Flushed Error: this connection's qp=262721 should be down while this WR=140296516013936 still in flight.
 -02-13 08:52:30.425907 7f99637fe700  1 RDMAStack handle_tx_event missing qp_num=262721 discard event
 -02-13 08:52:30.425908 7f99637fe700  1 RDMAStack handle_tx_event sending of the disconnect msg completed
 -02-13 08:52:30.425912 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f995402b1d0
 -02-13 08:52:30.425913 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f99540251f0
 -02-13 08:52:30.426340 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954000f70
 -02-13 08:52:30.426348 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f995402b330
 -02-13 08:52:30.426480 7f99637fe700 20 Infiniband rearm_notify started.
 -02-13 08:52:30.426485 7f99637fe700 20 Infiniband rearm_notify started.
       Cluster connection interrupted or timed out
 -02-13 08:52:30.456858 7f9962ffd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
 -02-13 08:52:30.456893 7f9962ffd700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
 -02-13 08:52:30.456903 7f9962ffd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
 -02-13 08:52:30.456910 7f9962ffd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f993400bc10
 -02-13 08:52:30.456922 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.456997 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
 -02-13 08:52:30.457043 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
 -02-13 08:52:30.457047 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
 -02-13 08:52:30.457502 7f9962ffd700 10 -- - shutdown -
 -02-13 08:52:30.457517 7f9962ffd700 10  Processor -- stop
 -02-13 08:52:30.457528 7f9962ffd700  1 -- - shutdown_connections
 -02-13 08:52:30.457531 7f9962ffd700  5 -- - shutdown_connections mark down 192.168.4.96:6789/0 0x7f9934008210
 -02-13 08:52:30.457544 7f9962ffd700  5 -- - shutdown_connections delete 0x7f9934008210
 -02-13 08:52:30.457557 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.457574 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.457590 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.457661 7f9962ffd700 10 -- - wait: waiting for dispatch queue
 -02-13 08:52:30.457710 7f9962ffd700 10 -- - wait: dispatch queue is stopped
 -02-13 08:52:30.457718 7f9962ffd700  1 -- - shutdown_connections
 -02-13 08:52:30.457723 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.457732 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.457738 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.457784 7f9962ffd700 10 -- - wait: done.
 -02-13 08:52:30.457791 7f9962ffd700  1 -- - wait complete.
 -02-13 08:52:30.457800 7f9962ffd700  1 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
 -02-13 08:52:30.457806 7f9962ffd700  2 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
 -02-13 08:52:30.457811 7f9962ffd700 10 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started
 -02-13 08:52:30.457818 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.458532 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.458607 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
 -02-13 08:52:30.458678 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
 -02-13 08:52:30.526605 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954008dd0
 -02-13 08:52:30.526612 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f9954038630
 -02-13 08:52:30.526838 7f9962ffd700 20 RDMAStack ~RDMADispatcher destructing rdma dispatcher

« Previous
1
2
3
Next »

(1-1/3)

Project

General

Profile

Ceph

Bug #22944 » ceph-client.smp-016.log