Project

General

Profile

ceph-client.smp-016.log

client log - Radosław Piliszek, 02/13/2018 08:01 AM

Download (46.4 KB)

 
1
2018-02-13 08:52:17.174866 7f9971b2d700  1 RDMAStack RDMAStack ms_async_rdma_enable_hugepage value is: 0
2
2018-02-13 08:52:17.174877 7f9971b2d700 20 RDMAStack RDMAStack constructing RDMAStack...
3
2018-02-13 08:52:17.174898 7f9971b2d700 20 RDMAStack  creating RDMAStack:0x7f996c0b54a8 with dispatcher:0x7f996c13e350
4
2018-02-13 08:52:17.175036 7f996b7fe700  2 Event(0x7f996c0b55a0 nevent=5000 time_id=1).set_owner idx=0 owner=140296910268160
5
2018-02-13 08:52:17.175039 7f996affd700  2 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).set_owner idx=1 owner=140296901875456
6
2018-02-13 08:52:17.175076 7f996b7fe700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).create_file_event create event started fd=4 mask=1 original mask is 0
7
2018-02-13 08:52:17.175073 7f99627fc700  2 Event(0x7f996c110840 nevent=5000 time_id=1).set_owner idx=2 owner=140296759265024
8
2018-02-13 08:52:17.175079 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=7 mask=1 original mask is 0
9
2018-02-13 08:52:17.175081 7f996b7fe700 20 EpollDriver.add_event add event fd=4 cur_mask=0 add_mask=1 to 3
10
2018-02-13 08:52:17.175084 7f996affd700 20 EpollDriver.add_event add event fd=7 cur_mask=0 add_mask=1 to 6
11
2018-02-13 08:52:17.175083 7f99627fc700 20 Event(0x7f996c110840 nevent=5000 time_id=1).create_file_event create event started fd=11 mask=1 original mask is 0
12
2018-02-13 08:52:17.175087 7f996b7fe700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).create_file_event create event end fd=4 mask=1 original mask is 1
13
2018-02-13 08:52:17.175088 7f99627fc700 20 EpollDriver.add_event add event fd=11 cur_mask=0 add_mask=1 to 10
14
2018-02-13 08:52:17.175090 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=7 mask=1 original mask is 1
15
2018-02-13 08:52:17.175093 7f996affd700 10 stack operator() starting
16
2018-02-13 08:52:17.175092 7f99627fc700 20 Event(0x7f996c110840 nevent=5000 time_id=1).create_file_event create event end fd=11 mask=1 original mask is 1
17
2018-02-13 08:52:17.175093 7f996b7fe700 10 stack operator() starting
18
2018-02-13 08:52:17.175096 7f99627fc700 10 stack operator() starting
19
2018-02-13 08:52:17.175323 7f9971b2d700 10 -- - ready -
20
2018-02-13 08:52:17.175330 7f9971b2d700  1  Processor -- start
21
2018-02-13 08:52:17.175385 7f9971b2d700  1 -- - start start
22
2018-02-13 08:52:17.176141 7f9971b2d700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
23
2018-02-13 08:52:17.176170 7f9971b2d700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
24
2018-02-13 08:52:17.176180 7f9971b2d700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).wakeup
25
2018-02-13 08:52:17.176191 7f9971b2d700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f996c154ec0
26
2018-02-13 08:52:17.176207 7f9971b2d700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
27
2018-02-13 08:52:17.176212 7f9971b2d700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).wakeup
28
2018-02-13 08:52:17.176208 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
29
2018-02-13 08:52:17.176227 7f9971b2d700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f996c152fc0 con 0
30
2018-02-13 08:52:17.176241 7f9971b2d700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f996c152fc0
31
2018-02-13 08:52:17.177693 7f996affd700  1 Infiniband binding_port found active port 1
32
2018-02-13 08:52:17.177728 7f996affd700  1 Infiniband init assigning: 1024 receive buffers
33
2018-02-13 08:52:17.177731 7f996affd700  1 Infiniband init assigning: 1024 send buffers
34
2018-02-13 08:52:17.177733 7f996affd700  1 Infiniband init device allow 4194303 completion entries
35
2018-02-13 08:52:17.250939 7f996affd700 20 Infiniband init started.
36
2018-02-13 08:52:17.250969 7f996affd700 20 Infiniband init started.
37
2018-02-13 08:52:17.252142 7f996affd700 20 Infiniband init successfully create cq=0x7f99540082b0
38
2018-02-13 08:52:17.253299 7f996affd700 20 Infiniband init successfully create cq=0x7f99540086a0
39
2018-02-13 08:52:17.253399 7f996affd700 20 Infiniband init started.
40
2018-02-13 08:52:17.253487 7f99637fe700 20 RDMAStack polling going to poll tx cq: 0x7f9954008280 rx cq: 0x7f9954008670
41
2018-02-13 08:52:17.253506 7f99637fe700 20 Infiniband rearm_notify started.
42
2018-02-13 08:52:17.253507 7f99637fe700 20 Infiniband rearm_notify started.
43
2018-02-13 08:52:17.253528 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f9954008f30
44
2018-02-13 08:52:17.253985 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f9954008f30
45
2018-02-13 08:52:17.254020 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
46
2018-02-13 08:52:17.255550 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
47
2018-02-13 08:52:17.255564 7f996affd700 10 Infiniband send_msg sending: 79, 262717, 0, 0, fe80000000000000f45214030035f771
48
2018-02-13 08:52:17.255604 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=18 mask=1 original mask is 0
49
2018-02-13 08:52:17.255608 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
50
2018-02-13 08:52:17.255613 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=18 mask=1 original mask is 1
51
2018-02-13 08:52:17.255618 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=17 mask=1 original mask is 0
52
2018-02-13 08:52:17.255619 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
53
2018-02-13 08:52:17.255621 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=17 mask=1 original mask is 1
54
2018-02-13 08:52:17.255624 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
55
2018-02-13 08:52:17.255642 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
56
2018-02-13 08:52:17.255669 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
57
2018-02-13 08:52:17.255675 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
58
2018-02-13 08:52:17.256391 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262717 tcp_fd: 18 notify_fd: 17
59
2018-02-13 08:52:17.256403 7f996affd700  5 Infiniband recv_msg recevd: 4, 0, 2790260736, 2140754944, ???j
60
2018-02-13 08:52:17.256406 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 0, 2790260736, 4, 2140754944> 
61
2018-02-13 08:52:17.256410 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
62
2018-02-13 08:52:17.256484 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
63
2018-02-13 08:52:17.256547 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
64
2018-02-13 08:52:17.256551 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954008dd0 with qp:0x7f9954008f30
65
2018-02-13 08:52:17.256553 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262717
66
2018-02-13 08:52:17.256557 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
67
2018-02-13 08:52:17.256563 7f996affd700 10 Infiniband send_msg sending: 79, 262717, 0, 0, fe80000000000000f45214030035f771
68
2018-02-13 08:52:17.256585 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
69
2018-02-13 08:52:17.256614 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
70
2018-02-13 08:52:17.256619 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
71
2018-02-13 08:52:17.256640 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262717
72
2018-02-13 08:52:17.256644 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
73
2018-02-13 08:52:17.256650 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
74
2018-02-13 08:52:17.256653 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262717 0x7f9954019700
75
2018-02-13 08:52:17.256674 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
76
2018-02-13 08:52:17.256731 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
77
2018-02-13 08:52:17.256733 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
78
2018-02-13 08:52:17.256739 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event started fd=17 mask=2 original mask is 1
79
2018-02-13 08:52:17.256741 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
80
2018-02-13 08:52:17.256743 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=1).create_file_event create event end fd=17 mask=2 original mask is 3
81
2018-02-13 08:52:17.256747 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
82
2018-02-13 08:52:17.256753 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
83
2018-02-13 08:52:17.256761 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262717 r = 8
84
2018-02-13 08:52:17.256766 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
85
2018-02-13 08:52:17.256771 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
86
2018-02-13 08:52:21.498227 7f99637fe700 20 RDMAStack polling got tx cq event.
87
2018-02-13 08:52:21.498238 7f99637fe700 20 RDMAStack polling tx completion queue got 1 responses.
88
2018-02-13 08:52:21.498245 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
89
2018-02-13 08:52:21.498250 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
90
2018-02-13 08:52:21.498265 7f99637fe700 20 Infiniband rearm_notify started.
91
2018-02-13 08:52:21.498266 7f99637fe700 20 Infiniband rearm_notify started.
92
2018-02-13 08:52:21.498306 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY
93
2018-02-13 08:52:21.498340 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262717 r = 8
94
2018-02-13 08:52:21.498345 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_bulk reading from fd=17 : Unknown error -104
95
2018-02-13 08:52:21.498368 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_until read failed
96
2018-02-13 08:52:21.498375 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection read banner and identify addresses failed
97
2018-02-13 08:52:21.498391 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
98
2018-02-13 08:52:21.498400 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
99
2018-02-13 08:52:21.498404 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
100
2018-02-13 08:52:21.498548 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).fault waiting 0.200000
101
2018-02-13 08:52:21.698850 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
102
2018-02-13 08:52:21.698888 7f996affd700 20 Infiniband init started.
103
2018-02-13 08:52:21.699048 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f995402b330
104
2018-02-13 08:52:21.699734 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f995402b330
105
2018-02-13 08:52:21.699755 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
106
2018-02-13 08:52:21.699925 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
107
2018-02-13 08:52:21.699943 7f996affd700 10 Infiniband send_msg sending: 79, 262719, 2116118, 0, fe80000000000000f45214030035f771
108
2018-02-13 08:52:21.699992 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=18 mask=1 original mask is 0
109
2018-02-13 08:52:21.700003 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
110
2018-02-13 08:52:21.700011 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=18 mask=1 original mask is 1
111
2018-02-13 08:52:21.700015 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=17 mask=1 original mask is 0
112
2018-02-13 08:52:21.700017 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
113
2018-02-13 08:52:21.700038 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=17 mask=1 original mask is 1
114
2018-02-13 08:52:21.700043 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
115
2018-02-13 08:52:21.700054 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
116
2018-02-13 08:52:21.701232 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262719 tcp_fd: 18 notify_fd: 17
117
2018-02-13 08:52:21.701270 7f996affd700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???j
118
2018-02-13 08:52:21.701275 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 0, 0, 4, 0> 
119
2018-02-13 08:52:21.701277 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
120
2018-02-13 08:52:21.701357 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
121
2018-02-13 08:52:21.701472 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
122
2018-02-13 08:52:21.701480 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954000f70 with qp:0x7f995402b330
123
2018-02-13 08:52:21.701483 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262719
124
2018-02-13 08:52:21.701493 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
125
2018-02-13 08:52:21.701508 7f996affd700 10 Infiniband send_msg sending: 79, 262719, 2116118, 0, fe80000000000000f45214030035f771
126
2018-02-13 08:52:21.701607 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
127
2018-02-13 08:52:21.701628 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
128
2018-02-13 08:52:21.701653 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
129
2018-02-13 08:52:21.701680 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262719
130
2018-02-13 08:52:21.701683 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
131
2018-02-13 08:52:21.701690 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
132
2018-02-13 08:52:21.701692 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262719 0x7f9954019700
133
2018-02-13 08:52:21.701697 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
134
2018-02-13 08:52:21.701770 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
135
2018-02-13 08:52:21.701775 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
136
2018-02-13 08:52:21.701785 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event started fd=17 mask=2 original mask is 1
137
2018-02-13 08:52:21.701789 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
138
2018-02-13 08:52:21.701793 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=2).create_file_event create event end fd=17 mask=2 original mask is 3
139
2018-02-13 08:52:21.701797 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
140
2018-02-13 08:52:21.701808 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
141
2018-02-13 08:52:21.701818 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262719 r = 8
142
2018-02-13 08:52:21.701828 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
143
2018-02-13 08:52:21.701835 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
144
2018-02-13 08:52:25.826826 7f99637fe700 20 RDMAStack polling got tx cq event.
145
2018-02-13 08:52:25.826835 7f99637fe700 20 RDMAStack polling tx completion queue got 1 responses.
146
2018-02-13 08:52:25.826837 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
147
2018-02-13 08:52:25.826839 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
148
2018-02-13 08:52:25.826845 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954008dd0
149
2018-02-13 08:52:25.826847 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f9954008f30
150
2018-02-13 08:52:25.826908 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY
151
2018-02-13 08:52:25.826939 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262719 r = 8
152
2018-02-13 08:52:25.826944 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_bulk reading from fd=17 : Unknown error -104
153
2018-02-13 08:52:25.826956 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).read_until read failed
154
2018-02-13 08:52:25.826962 7f996affd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection read banner and identify addresses failed
155
2018-02-13 08:52:25.826969 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
156
2018-02-13 08:52:25.826978 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
157
2018-02-13 08:52:25.826980 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
158
2018-02-13 08:52:25.827137 7f99637fe700 10 RDMAStack handle_async_event event associated qp=0x7f995402b330 evt: last WQE reached
159
2018-02-13 08:52:25.827145 7f99637fe700  1 RDMAStack handle_async_event it's not forwardly stopped by us, reenable=0x7f995402b0c0
160
2018-02-13 08:52:25.827147 7f99637fe700  1  RDMAConnectedSocketImpl fault tcp fd 18
161
2018-02-13 08:52:25.827151 7f99637fe700 20 Infiniband rearm_notify started.
162
2018-02-13 08:52:25.827153 7f99637fe700 20 Infiniband rearm_notify started.
163
2018-02-13 08:52:25.827165 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954000f70
164
2018-02-13 08:52:25.827168 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f995402b330
165
2018-02-13 08:52:25.827235 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).fault waiting 0.400000
166
2018-02-13 08:52:26.227711 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
167
2018-02-13 08:52:26.227757 7f996affd700 20 Infiniband init started.
168
2018-02-13 08:52:26.227918 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f995402b330
169
2018-02-13 08:52:26.228564 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f995402b330
170
2018-02-13 08:52:26.228592 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
171
2018-02-13 08:52:26.228764 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
172
2018-02-13 08:52:26.228783 7f996affd700 10 Infiniband send_msg sending: 79, 262721, 5515815, 0, fe80000000000000f45214030035f771
173
2018-02-13 08:52:26.228830 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
174
2018-02-13 08:52:26.228842 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
175
2018-02-13 08:52:26.228849 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
176
2018-02-13 08:52:26.228853 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
177
2018-02-13 08:52:26.228856 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
178
2018-02-13 08:52:26.228860 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
179
2018-02-13 08:52:26.228863 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
180
2018-02-13 08:52:26.228875 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
181
2018-02-13 08:52:26.229831 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262721 tcp_fd: 18 notify_fd: 17
182
2018-02-13 08:52:26.229869 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754944, 2258632704, 2140754944, ???j
183
2018-02-13 08:52:26.229874 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754944, 2258632704, 4, 2140754944> 
184
2018-02-13 08:52:26.229876 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
185
2018-02-13 08:52:26.229970 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
186
2018-02-13 08:52:26.230021 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
187
2018-02-13 08:52:26.230029 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954000f70 with qp:0x7f995402b330
188
2018-02-13 08:52:26.230032 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262721
189
2018-02-13 08:52:26.230034 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
190
2018-02-13 08:52:26.230048 7f996affd700 10 Infiniband send_msg sending: 79, 262721, 5515815, 2140754944, fe80000000000000f45214030035f771
191
2018-02-13 08:52:26.230109 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
192
2018-02-13 08:52:26.230129 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
193
2018-02-13 08:52:26.230146 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
194
2018-02-13 08:52:26.230172 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262721
195
2018-02-13 08:52:26.230180 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
196
2018-02-13 08:52:26.230187 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
197
2018-02-13 08:52:26.230190 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262721 0x7f9954019700
198
2018-02-13 08:52:26.230194 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
199
2018-02-13 08:52:26.230248 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
200
2018-02-13 08:52:26.230252 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
201
2018-02-13 08:52:26.230261 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
202
2018-02-13 08:52:26.230265 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
203
2018-02-13 08:52:26.230269 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
204
2018-02-13 08:52:26.230272 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
205
2018-02-13 08:52:26.230283 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
206
2018-02-13 08:52:26.230293 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262721 r = 8
207
2018-02-13 08:52:26.230303 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
208
2018-02-13 08:52:26.230310 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
209
2018-02-13 08:52:27.176292 7f9968ff9700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
210
2018-02-13 08:52:27.176315 7f9968ff9700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
211
2018-02-13 08:52:27.176329 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
212
2018-02-13 08:52:27.176341 7f9968ff9700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f996c154ec0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f996c152fc0
213
2018-02-13 08:52:27.176365 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
214
2018-02-13 08:52:27.176413 7f9968ff9700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
215
2018-02-13 08:52:27.176437 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
216
2018-02-13 08:52:27.176488 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
217
2018-02-13 08:52:27.176492 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
218
2018-02-13 08:52:27.176513 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
219
2018-02-13 08:52:27.176534 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
220
2018-02-13 08:52:27.176544 7f9968ff9700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f99340013d0
221
2018-02-13 08:52:27.176570 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
222
2018-02-13 08:52:27.176595 7f9968ff9700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9934006040 con 0
223
2018-02-13 08:52:27.176594 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
224
2018-02-13 08:52:27.176612 7f9968ff9700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f9934006040
225
2018-02-13 08:52:27.176620 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
226
2018-02-13 08:52:27.176621 7f996affd700 20 Infiniband init started.
227
2018-02-13 08:52:27.176788 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f99540251f0
228
2018-02-13 08:52:27.177307 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f99540251f0
229
2018-02-13 08:52:27.177327 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
230
2018-02-13 08:52:27.177512 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
231
2018-02-13 08:52:27.177531 7f996affd700 10 Infiniband send_msg sending: 79, 262723, 10238434, 0, fe80000000000000f45214030035f771
232
2018-02-13 08:52:27.177578 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
233
2018-02-13 08:52:27.177589 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
234
2018-02-13 08:52:27.177596 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
235
2018-02-13 08:52:27.177600 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
236
2018-02-13 08:52:27.177602 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
237
2018-02-13 08:52:27.177606 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
238
2018-02-13 08:52:27.177610 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
239
2018-02-13 08:52:27.177622 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
240
2018-02-13 08:52:27.177630 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
241
2018-02-13 08:52:27.177648 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
242
2018-02-13 08:52:27.178651 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262723 tcp_fd: 18 notify_fd: 17
243
2018-02-13 08:52:27.178675 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754944, 2258632704, 2140754944, ???j
244
2018-02-13 08:52:27.178698 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754944, 2258632704, 4, 2140754944> 
245
2018-02-13 08:52:27.178702 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
246
2018-02-13 08:52:27.178760 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
247
2018-02-13 08:52:27.178835 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
248
2018-02-13 08:52:27.178842 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f995402b1d0 with qp:0x7f99540251f0
249
2018-02-13 08:52:27.178845 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262723
250
2018-02-13 08:52:27.178850 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
251
2018-02-13 08:52:27.178862 7f996affd700 10 Infiniband send_msg sending: 79, 262723, 10238434, 2140754944, fe80000000000000f45214030035f771
252
2018-02-13 08:52:27.178908 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
253
2018-02-13 08:52:27.178925 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
254
2018-02-13 08:52:27.178932 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
255
2018-02-13 08:52:27.178959 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262723
256
2018-02-13 08:52:27.178962 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
257
2018-02-13 08:52:27.178968 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
258
2018-02-13 08:52:27.178972 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262723 0x7f99540196e0
259
2018-02-13 08:52:27.178977 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
260
2018-02-13 08:52:27.179045 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
261
2018-02-13 08:52:27.179051 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
262
2018-02-13 08:52:27.179066 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
263
2018-02-13 08:52:27.179071 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
264
2018-02-13 08:52:27.179075 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
265
2018-02-13 08:52:27.179078 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
266
2018-02-13 08:52:27.179089 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
267
2018-02-13 08:52:27.179099 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262723 r = 8
268
2018-02-13 08:52:27.179108 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
269
2018-02-13 08:52:27.179116 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
270
2018-02-13 08:52:30.176830 7f9968ff9700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
271
2018-02-13 08:52:30.176843 7f9968ff9700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
272
2018-02-13 08:52:30.176861 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
273
2018-02-13 08:52:30.176867 7f9968ff9700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f99340013d0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f9934006040
274
2018-02-13 08:52:30.176881 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
275
2018-02-13 08:52:30.176909 7f9968ff9700 10 -- - create_connect 192.168.4.96:6789/0, creating connection and registering
276
2018-02-13 08:52:30.176938 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
277
2018-02-13 08:52:30.176946 7f9968ff9700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
278
2018-02-13 08:52:30.176943 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
279
2018-02-13 08:52:30.176953 7f9968ff9700 10 -- - get_connection mon.0 192.168.4.96:6789/0 new 0x7f9934008210
280
2018-02-13 08:52:30.176961 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
281
2018-02-13 08:52:30.176971 7f9968ff9700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
282
2018-02-13 08:52:30.176964 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
283
2018-02-13 08:52:30.176988 7f9968ff9700  1 -- - --> 192.168.4.96:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f993400bc10 con 0
284
2018-02-13 08:52:30.177002 7f9968ff9700 15 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f993400bc10
285
2018-02-13 08:52:30.177067 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
286
2018-02-13 08:52:30.177096 7f996affd700 20 Infiniband init started.
287
2018-02-13 08:52:30.177273 7f996affd700 20 Infiniband init successfully create queue pair: qp=0x7f9954038630
288
2018-02-13 08:52:30.177885 7f996affd700 20 Infiniband init successfully change queue pair to INIT: qp=0x7f9954038630
289
2018-02-13 08:52:30.177906 7f996affd700 20  RDMAConnectedSocketImpl try_connect nonblock:1, nodelay:1, rbuf_size: 0
290
2018-02-13 08:52:30.178068 7f996affd700 20  RDMAConnectedSocketImpl try_connect tcp_fd: 18
291
2018-02-13 08:52:30.178086 7f996affd700 10 Infiniband send_msg sending: 79, 262725, 11225430, 0, fe80000000000000f45214030035f771
292
2018-02-13 08:52:30.178132 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=18 mask=1 original mask is 0
293
2018-02-13 08:52:30.178143 7f996affd700 20 EpollDriver.add_event add event fd=18 cur_mask=0 add_mask=1 to 6
294
2018-02-13 08:52:30.178150 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=18 mask=1 original mask is 1
295
2018-02-13 08:52:30.178154 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=1 original mask is 0
296
2018-02-13 08:52:30.178157 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=0 add_mask=1 to 6
297
2018-02-13 08:52:30.178160 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=1 original mask is 1
298
2018-02-13 08:52:30.178178 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING
299
2018-02-13 08:52:30.178191 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
300
2018-02-13 08:52:30.178199 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
301
2018-02-13 08:52:30.178205 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
302
2018-02-13 08:52:30.178996 7f996affd700 20  RDMAConnectedSocketImpl handle_connection QP: 262725 tcp_fd: 18 notify_fd: 17
303
2018-02-13 08:52:30.179018 7f996affd700  5 Infiniband recv_msg recevd: 4, 2140754947, 0, 0, ???j
304
2018-02-13 08:52:30.179021 7f996affd700 20  RDMAConnectedSocketImpl handle_connection peer msg :  < 2140754947, 0, 4, 0> 
305
2018-02-13 08:52:30.179022 7f996affd700 20  RDMAConnectedSocketImpl activate Choosing gid_index 52, sl 3
306
2018-02-13 08:52:30.179070 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
307
2018-02-13 08:52:30.179127 7f996affd700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
308
2018-02-13 08:52:30.179131 7f996affd700 20  RDMAConnectedSocketImpl activate QueuePair: 0x7f9954008dd0 with qp:0x7f9954038630
309
2018-02-13 08:52:30.179133 7f996affd700 20  RDMAConnectedSocketImpl activate handle fake send, wake it up. QP: 262725
310
2018-02-13 08:52:30.179138 7f996affd700 20  RDMAConnectedSocketImpl submit we need 0 bytes. iov size: 0
311
2018-02-13 08:52:30.179147 7f996affd700 10 Infiniband send_msg sending: 79, 262725, 11225430, 2140754947, fe80000000000000f45214030035f771
312
2018-02-13 08:52:30.179220 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
313
2018-02-13 08:52:30.179239 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=1 delmask=2 to 6
314
2018-02-13 08:52:30.179247 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
315
2018-02-13 08:52:30.179286 7f996affd700 20  RDMAConnectedSocketImpl send QP: 262725
316
2018-02-13 08:52:30.179294 7f996affd700 20  RDMAConnectedSocketImpl submit we need 9 bytes. iov size: 1
317
2018-02-13 08:52:30.179301 7f996affd700 20  RDMAConnectedSocketImpl submit left bytes: 0 in buffers 0 tx chunks 1
318
2018-02-13 08:52:30.179303 7f996affd700 20  RDMAConnectedSocketImpl post_work_request QP: 262725 0x7f99540196c0
319
2018-02-13 08:52:30.179308 7f996affd700 20  RDMAConnectedSocketImpl post_work_request qp state is IBV_QPS_RTS
320
2018-02-13 08:52:30.179364 7f996affd700 20  RDMAConnectedSocketImpl submit finished sending 9 bytes.
321
2018-02-13 08:52:30.179368 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
322
2018-02-13 08:52:30.179378 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event started fd=17 mask=2 original mask is 1
323
2018-02-13 08:52:30.179381 7f996affd700 20 EpollDriver.add_event add event fd=17 cur_mask=1 add_mask=2 to 6
324
2018-02-13 08:52:30.179386 7f996affd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).create_file_event create event end fd=17 mask=2 original mask is 3
325
2018-02-13 08:52:30.179389 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 192.168.4.96:6789/0
326
2018-02-13 08:52:30.179400 7f996affd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).process prev state is STATE_CONNECTING_RE
327
2018-02-13 08:52:30.179410 7f996affd700 20  RDMAConnectedSocketImpl read notify_fd : 1 in 262725 r = 8
328
2018-02-13 08:52:30.179419 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
329
2018-02-13 08:52:30.179426 7f996affd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
330
2018-02-13 08:52:30.425890 7f99637fe700 20 RDMAStack polling got tx cq event.
331
2018-02-13 08:52:30.425901 7f99637fe700 20 RDMAStack polling tx completion queue got 2 responses.
332
2018-02-13 08:52:30.425903 7f99637fe700  1 RDMAStack handle_tx_event connection between server and client not working. Disconnect this now
333
2018-02-13 08:52:30.425904 7f99637fe700  1 RDMAStack handle_tx_event missing qp_num=262721 discard event
334
2018-02-13 08:52:30.425905 7f99637fe700  1 RDMAStack handle_tx_event Work Request Flushed Error: this connection's qp=262721 should be down while this WR=140296516013936 still in flight.
335
2018-02-13 08:52:30.425907 7f99637fe700  1 RDMAStack handle_tx_event missing qp_num=262721 discard event
336
2018-02-13 08:52:30.425908 7f99637fe700  1 RDMAStack handle_tx_event sending of the disconnect msg completed
337
2018-02-13 08:52:30.425912 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f995402b1d0
338
2018-02-13 08:52:30.425913 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f99540251f0
339
2018-02-13 08:52:30.426340 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954000f70
340
2018-02-13 08:52:30.426348 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f995402b330
341
2018-02-13 08:52:30.426480 7f99637fe700 20 Infiniband rearm_notify started.
342
2018-02-13 08:52:30.426485 7f99637fe700 20 Infiniband rearm_notify started.
343
Cluster connection interrupted or timed out
344
2018-02-13 08:52:30.456858 7f9962ffd700  1 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).mark_down
345
2018-02-13 08:52:30.456893 7f9962ffd700  2 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._stop
346
2018-02-13 08:52:30.456903 7f9962ffd700 10 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue started
347
2018-02-13 08:52:30.456910 7f9962ffd700 20 -- - >> 192.168.4.96:6789/0 conn(0x7f9934008210 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).discard_out_queue discard 0x7f993400bc10
348
2018-02-13 08:52:30.456922 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
349
2018-02-13 08:52:30.456997 7f996affd700 20 EpollDriver.del_event del event fd=17 cur_mask=3 delmask=3 to 6
350
2018-02-13 08:52:30.457043 7f996affd700 20  RDMAConnectedSocketImpl ~RDMAConnectedSocketImpl destruct.
351
2018-02-13 08:52:30.457047 7f996affd700 20 EpollDriver.del_event del event fd=18 cur_mask=1 delmask=1 to 6
352
2018-02-13 08:52:30.457502 7f9962ffd700 10 -- - shutdown -
353
2018-02-13 08:52:30.457517 7f9962ffd700 10  Processor -- stop
354
2018-02-13 08:52:30.457528 7f9962ffd700  1 -- - shutdown_connections 
355
2018-02-13 08:52:30.457531 7f9962ffd700  5 -- - shutdown_connections mark down 192.168.4.96:6789/0 0x7f9934008210
356
2018-02-13 08:52:30.457544 7f9962ffd700  5 -- - shutdown_connections delete 0x7f9934008210
357
2018-02-13 08:52:30.457557 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
358
2018-02-13 08:52:30.457574 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
359
2018-02-13 08:52:30.457590 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
360
2018-02-13 08:52:30.457661 7f9962ffd700 10 -- - wait: waiting for dispatch queue
361
2018-02-13 08:52:30.457710 7f9962ffd700 10 -- - wait: dispatch queue is stopped
362
2018-02-13 08:52:30.457718 7f9962ffd700  1 -- - shutdown_connections 
363
2018-02-13 08:52:30.457723 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
364
2018-02-13 08:52:30.457732 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
365
2018-02-13 08:52:30.457738 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
366
2018-02-13 08:52:30.457784 7f9962ffd700 10 -- - wait: done.
367
2018-02-13 08:52:30.457791 7f9962ffd700  1 -- - wait complete.
368
2018-02-13 08:52:30.457800 7f9962ffd700  1 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
369
2018-02-13 08:52:30.457806 7f9962ffd700  2 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
370
2018-02-13 08:52:30.457811 7f9962ffd700 10 -- - >> - conn(0x7f996c13ffd0 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started
371
2018-02-13 08:52:30.457818 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
372
2018-02-13 08:52:30.458532 7f9962ffd700 20 Event(0x7f996c0b55a0 nevent=5000 time_id=1).wakeup
373
2018-02-13 08:52:30.458607 7f9962ffd700 20 Event(0x7f996c0e2ef0 nevent=5000 time_id=3).wakeup
374
2018-02-13 08:52:30.458678 7f9962ffd700 20 Event(0x7f996c110840 nevent=5000 time_id=1).wakeup
375
2018-02-13 08:52:30.526605 7f99637fe700 10 RDMAStack polling finally delete qp=0x7f9954008dd0
376
2018-02-13 08:52:30.526612 7f99637fe700 20 Infiniband ~QueuePair destroy qp=0x7f9954038630
377
2018-02-13 08:52:30.526838 7f9962ffd700 20 RDMAStack ~RDMADispatcher destructing rdma dispatcher