Bug #20049: msg: libraries are binding to incorrect client address - Ceph - Ceph

Actions

Copy link

Bug #20049

closed

msg: libraries are binding to incorrect client address

Added by Jason Dillaman almost 7 years ago. Updated almost 3 years ago.

Status:

Closed

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

librados clients running within a VM with QEMU user networking (e.g. ip address 10.0.2.15) are incorrectly attempting to bind to the hypervisor IP address. Appears to be related to changes introduced in https://github.com/ceph/ceph/pull/12901

2017-05-23 02:50:39.717415 7ffff7fc1c40 10 -- - create_connect 172.21.15.160:6789/0, creating connection and registering
2017-05-23 02:50:39.717465 7ffff7fc1c40 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
2017-05-23 02:50:39.717503 7ffff7fc1c40 10 -- - get_connection mon.0 172.21.15.160:6789/0 new 0x555555ae77b0
2017-05-23 02:50:39.717521 7ffff7fc1c40 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_keepalive
2017-05-23 02:50:39.717539 7ffff7fc1c40  1 -- - --> 172.21.15.160:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x555555ae3aa0 con 0
2017-05-23 02:50:39.726001 7fffe9118700 10 NetHandler reconnect reconnect: Operation already in progress
2017-05-23 02:50:39.726026 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection nonblock connect inprogress
2017-05-23 02:50:39.726047 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
2017-05-23 02:50:39.726062 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1).handle_write
2017-05-23 02:50:39.726121 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._process_connection connect successfully, ready to send banner
2017-05-23 02:50:39.726223 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=1)._try_send sent bytes 9 remaining bytes 0
2017-05-23 02:50:39.726237 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._process_connection connect write banner done: 172.21.15.160:6789/0
2017-05-23 02:50:39.726253 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_write
2017-05-23 02:50:39.726257 7fffe9118700 10 -- - >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
2017-05-23 02:50:39.726423 7fffe9118700  1 -- 172.21.15.71:0/4228153022 learned_addr learned my addr 172.21.15.71:0/4228153022
2017-05-23 02:50:39.726510 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1)._try_send sent bytes 136 remaining bytes 0
2017-05-23 02:50:39.726533 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1)._process_connection connect sent my addr 172.21.15.71:0/4228153022
2017-05-23 02:50:39.726544 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1)._process_connection connect sending gseq=1 cseq=0 proto=15
2017-05-23 02:50:39.726591 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1)._try_send sent bytes 33 remaining bytes 0
2017-05-23 02:50:39.726606 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=0 cs=0 l=1).handle_write
2017-05-23 02:50:39.726610 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=0 cs=0 l=1)._try_send sent bytes 0 remaining bytes 0
2017-05-23 02:50:39.733947 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=1).handle_connect_reply got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq
2017-05-23 02:50:39.733968 7fffe9118700  2 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
2017-05-23 02:50:39.733981 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1).discard_requeued_up_to 0
2017-05-23 02:50:39.734097 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._try_send sent bytes 8 remaining bytes 0
2017-05-23 02:50:39.734118 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_CONNECTING_READY pgs=0 cs=0 l=1)._process_connection send in_seq done 
2017-05-23 02:50:39.734130 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._process_connection connect success 1, lossy = 1, features 1152323339925389307
2017-05-23 02:50:39.734177 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.734189 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._append_keepalive_or_ack
2017-05-23 02:50:39.734314 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 144 remaining bytes 0
2017-05-23 02:50:39.734337 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x555555ae3aa0 done.
2017-05-23 02:50:39.734356 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.774969 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
2017-05-23 02:50:39.774998 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
2017-05-23 02:50:39.775005 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 1 0x7fffd8003e70 mon_map magic: 0 v1[New Thread 0x7fffd17fa700 (LWP 1667)]

2017-05-23 02:50:39.775030 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
[New Thread 0x7fffd0ff9700 (LWP 1668)]
2017-05-23 02:50:39.775038 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
[New Thread 0x7fffcbfff700 (LWP 1669)]
2017-05-23 02:50:39.775044 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 2 0x7fffd8004410 auth_reply(proto 2 0 (0) Success) v1
2017-05-23 02:50:39.775119 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 1 ==== mon_map magic: 0 v1 ==== 173+0+0 (1811183145 0 0) 0x7fffd8003e70 con 0x555555ae77b0
2017-05-23 02:50:39.775179 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 173 to dispatch throttler 206/104857600
2017-05-23 02:50:39.775188 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (397928211 0 0) 0x7fffd8004410 con 0x555555ae77b0
2017-05-23 02:50:39.779572 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 --> 172.21.15.160:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fffcc001550 con 0
2017-05-23 02:50:39.779600 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 33 to dispatch throttler 33/104857600
2017-05-23 02:50:39.779622 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.779743 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 137 remaining bytes 0
2017-05-23 02:50:39.779757 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x7fffcc001550 done.
2017-05-23 02:50:39.780697 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
2017-05-23 02:50:39.780710 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
2017-05-23 02:50:39.780716 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 3 0x7fffd8004b70 auth_reply(proto 2 0 (0) Success) v1
2017-05-23 02:50:39.780749 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2289509696 0 0) 0x7fffd8004b70 con 0x555555ae77b0
2017-05-23 02:50:39.780860 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 --> 172.21.15.160:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fffcc001f70 con 0
2017-05-23 02:50:39.780875 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 206 to dispatch throttler 206/104857600
2017-05-23 02:50:39.780891 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.780997 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 270 remaining bytes 0
2017-05-23 02:50:39.781010 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x7fffcc001f70 done.
2017-05-23 02:50:39.782272 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
2017-05-23 02:50:39.782287 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
2017-05-23 02:50:39.782292 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 4 0x7fffd8005b70 auth_reply(proto 2 0 (0) Success) v1
2017-05-23 02:50:39.782324 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (1666957310 0 0) 0x7fffd8005b70 con 0x555555ae77b0
2017-05-23 02:50:39.782453 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 --> 172.21.15.160:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x555555ae3ee0 con 0
2017-05-23 02:50:39.782477 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 580 to dispatch throttler 580/104857600
2017-05-23 02:50:39.782492 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.782602 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 98 remaining bytes 0
2017-05-23 02:50:39.782615 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x555555ae3ee0 done.
2017-05-23 02:50:39.782659 7ffff7fc1c40  1 -- 172.21.15.71:0/4228153022 --> 172.21.15.160:6789/0 -- mon_subscribe({mgrmap=0+}) v2 -- 0x555555ae3ee0 con 0
2017-05-23 02:50:39.782679 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.782709 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 98 remaining bytes 0
2017-05-23 02:50:39.782715 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x555555ae3ee0 done.
2017-05-23 02:50:39.784272 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
2017-05-23 02:50:39.784290 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
2017-05-23 02:50:39.784296 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 5 0x7fffd80064a0 mon_map magic: 0 v1
2017-05-23 02:50:39.784320 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 5 ==== mon_map magic: 0 v1 ==== 173+0+0 (1811183145 0 0) 0x7fffd80064a0 con 0x555555ae77b0
2017-05-23 02:50:39.784362 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 173 to dispatch throttler 173/104857600
2017-05-23 02:50:39.784439 7ffff7fc1c40  1 -- 172.21.15.71:0/4228153022 --> 172.21.15.160:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x555555ae43b0 con 0
2017-05-23 02:50:39.784459 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).handle_write
2017-05-23 02:50:39.784593 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1)._try_send sent bytes 98 remaining bytes 0
2017-05-23 02:50:39.784606 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN pgs=20 cs=1 l=1).write_message sending 0x555555ae43b0 done.
2017-05-23 02:50:39.784745 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process aborted = 0
2017-05-23 02:50:39.784762 7fffe9118700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1).process no session security set
2017-05-23 02:50:39.784767 7fffe9118700  5 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6789/0 conn(0x555555ae77b0 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx mon.0 seq 6 0x7fffd8006ca0 mgrmap(e 3) v1
2017-05-23 02:50:39.784790 7fffd37fe700  1 -- 172.21.15.71:0/4228153022 <== mon.0 172.21.15.160:6789/0 6 ==== mgrmap(e 3) v1 ==== 63+0+0 (2279788570 0 0) 0x7fffd8006ca0 con 0x555555ae77b0
2017-05-23 02:50:39.784820 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 create_connect 172.21.15.160:6800/8897, creating connection and registering
2017-05-23 02:50:39.784841 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 >> 172.21.15.160:6800/8897 conn(0x7fffcc009e60 :-1 s=STATE_NONE pgs=0 cs=0 l=1)._connect csq=0
2017-05-23 02:50:39.784865 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 get_connection mgr.4098 172.21.15.160:6800/8897 new 0x7fffcc009e60
2017-05-23 02:50:39.784882 7fffd37fe700 10 -- 172.21.15.71:0/4228153022 dispatch_throttle_release 63 to dispatch throttler 63/104857600
2017-05-23 02:50:39.784909 7fffe8917700  2 NetHandler generic_connect client bind error , (99) Cannot assign requested address

Actions

Copy link

Updated by Haomai Wang almost 7 years ago

do you think set "ms_bind_before_connect = false" is the expected solution?

Actions

Copy link

Updated by Jason Dillaman almost 7 years ago

@Haomai: I don't really understand why we would want clients to implicitly bind to a learned address. I am using that config option to get the OpenStack devstack tests functional again, but this seems like an unintended consequence. The fact that it first connects to a monitor w/o binding and then fails to connect to OSDs since it learned its address from a monitor just seems a bit off. Plus, it's not even logged as an error when the bind fails so I spend a few hours trying to diagnose this since I didn't have direct access to the environment where it was failing.

Actions

Copy link

Updated by Haomai Wang almost 7 years ago

I guess https://github.com/ceph/ceph/pull/7256 could give a more precise description about the fix.

I propose a pull request(https://github.com/ceph/ceph/pull/15243) to fix your problem. Then I think we need to think about how to learn this instead of config value....

Actions

Copy link