Project

General

Profile

Actions
Copy link

Bug #49534

closed

osd connection loop

Added by Sage Weil about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
39739
Crash signature (v1):
Crash signature (v2):

Description

osd.3 is trying to connection to osd.0 and getting disconnected, then retrying, in a loop

here is a sample attempt and rejection:

osd.3:


2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 create_connect v2:172.21.15.131:6801/7, creating connection and registering
2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_NONE l=1)._connect
2021-02-27T16:10:00.376+0000 7f4d25fd1700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
2021-02-27T16:10:00.376+0000 7f4d25fd1700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055689).wakeup
2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 create_connect 0x55f5466c1c00 v2:172.21.15.131:6801/7 v2:172.21.15.131:6801/7
2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 connect_to v2:172.21.15.131:6801/7 new 0x55f5466c1c00
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTING l=1).process
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055690).create_file_event create event started fd=73 mask=1 original mask is 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 EpollDriver.add_event add event fd=73 cur_mask=0 add_mask=1 to 5
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055690).create_file_event create event end fd=73 mask=1 current mask is 1
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 EpollDriver.del_event del event fd=73 cur_mask=1 delmask=2 to 5
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTING_RE l=1).process connect successfully, ready to send banner
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).read_event
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).start_client_banner_exchange
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._banner_exchange
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 26 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=10
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=10
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=16
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 72 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=40
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=1
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello payload.length()=36
2021-02-27T16:10:00.376+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello received hello: peer_type=4 peer_addr_for_me=v2:172.21.15.131:57564/0
2021-02-27T16:10:00.376+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello getsockname says I am 172.21.15.131:57564 when talking to v2:172.21.15.131:6801/7
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).post_client_banner_exchange
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request peer_type 4 auth_client 0x7ffee9de5900
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 230 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=40
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=4
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more payload.length()=36
2021-02-27T16:10:00.376+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more auth reply more len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 214 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=56
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=6
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done payload.length()=52
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 68 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=36
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=7
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_signature payload.length()=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_signature pre-auth signature success sig_frame.signature()=68cd43d6ef51988f2765412e0dca3701bef36a9917153a20923ff3d262d5d231
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident
2021-02-27T16:10:00.376+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252313 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 159 remaining bytes 0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=92
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=9
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident payload.length()=88
2021-02-27T16:10:00.376+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident received server identification: addrs=v2:172.21.15.131:6801/7 gid=0 global_seq=6505107 features_supported=3f01cfbb7ffdffff features_required=e01020042240000 flags=1 cookie=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident connect success 0, lossy = 1, features 4540138303579357183
2021-02-27T16:10:00.376+0000 7f4d420cb700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
2021-02-27T16:10:00.377+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process
2021-02-27T16:10:00.377+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32
2021-02-27T16:10:00.377+0000 7f4d420cb700  1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 73
2021-02-27T16:10:00.377+0000 7f4d420cb700  1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed
2021-02-27T16:10:00.377+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=-1
2021-02-27T16:10:00.377+0000 7f4d420cb700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
2021-02-27T16:10:00.377+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0)._fault
2021-02-27T16:10:00.377+0000 7f4d420cb700  2 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0)._fault on lossy channel, failing
2021-02-27T16:10:00.377+0000 7f4d420cb700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).stop
2021-02-27T16:10:00.377+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state
2021-02-27T16:10:00.377+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).reset_security
2021-02-27T16:10:00.377+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started
2021-02-27T16:10:00.377+0000 7f4d420cb700 20 EpollDriver.del_event del event fd=73 cur_mask=1 delmask=3 to 5
2021-02-27T16:10:00.377+0000 7f4d25fd1700  1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CLOSED l=1).mark_down
2021-02-27T16:10:00.377+0000 7f4d25fd1700  1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=CLOSED pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).stop

on osd.0,

2021-02-27T16:10:00.376+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252313 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._infer_target_addr 172.21.15.131:0/7 -> 172.21.15.131:0/7
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b37dc00
2021-02-27T16:10:00.376+0000 7fb0f6a06700  1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b37dc00 is a lossy channel. Stopping existing in favor of this connection
2021-02-27T16:10:00.376+0000 7fb0f6a06700  1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).stop
2021-02-27T16:10:00.376+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 Event(0x559f539754c0 nevent=5000 time_id=4082214).wakeup
2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 Event(0x559f539754c0 nevent=5000 time_id=4082214).wakeup
2021-02-27T16:10:00.376+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=CLOSED pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state (warped) reseting crypto handlers
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).send_server_ident
2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).discard_requeued_up_to 0
2021-02-27T16:10:00.376+0000 7fb0e5121700  2 osd.0 1419 ms_handle_reset con 0x559f6b37dc00 session 0x559f6a106d20
2021-02-27T16:10:00.376+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).send_server_ident sending identification: addrs=v2:172.21.15.131:6801/7 gid=0 global_seq=6505106 features_supported=3f01cfbb7ffdffff features_required=e01020042240000 flags=1 cookie=0
2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 accept_conn 0x559f6b930c00 172.21.15.131:0/7
2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 124 remaining bytes 0
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).server_ready
2021-02-27T16:10:00.376+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=CLOSED pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_security
2021-02-27T16:10:00.376+0000 7fb0f6a06700  1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame
2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32
...
2021-02-27T16:10:00.376+0000 7fb0f5a04700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 msgr2=0x559f6c184a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._infer_target_addr 172.21.15.131:0/7 -> 172.21.15.131:0/7
2021-02-27T16:10:00.376+0000 7fb0f5a04700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 0x559f6c184a00 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b930c00
2021-02-27T16:10:00.376+0000 7fb0f5a04700  1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 0x559f6c184a00 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b930c00 is a lossy channel. Stopping existing in favor of this connection
2021-02-27T16:10:00.376+0000 7fb0f5a04700  1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).stop
2021-02-27T16:10:00.376+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state
2021-02-27T16:10:00.376+0000 7fb0f5a04700 20 Event(0x559f53975040 nevent=5000 time_id=2045227).wakeup
2021-02-27T16:10:00.376+0000 7fb0f5a04700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started

i.e., we are disconnected because another connection comes in from the same peer. that second connection has the same seq!

this is reproduced by rados/cephadm/thrash with podman --init arg enabled.

Related issues 1 (1 open — 0 closed)

Related to RADOS - Bug #57977: osd:tick checking mon for new mapPending BackportRadoslaw Zarzynski

Actions
Actions
Copy link
 #1

Updated by Sage Weil about 3 years ago

  • Description updated (diff)

on osd.3,

2021-02-27T16:10:00.463+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f53dc81c00 0x55f53c0e9400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252374 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.464+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252375 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.465+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f53dc81c00 0x55f53c0e9400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252376 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.467+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252377 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.468+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f53e616400 0x55f53e7ff400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252378 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.469+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252379 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.470+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f53e616400 0x55f53e7ff400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252380 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0

notice global_seq increases with each attempt

on osd.0,

2021-02-27T16:10:00.426+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252350 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.426+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f657de800 0x559f6b8bef00 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252350 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.427+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252351 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.427+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f62af8000 0x559f65528500 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252351 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.428+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252352 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.429+0000 7fb0f5a04700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f65491c00 0x559f6b3b6300 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252352 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0
2021-02-27T16:10:00.429+0000 7fb0f6a06700  5 --2- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252353 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0

where each seq appears 2x

Actions
Copy link
 #2

Updated by Sage Weil about 3 years ago 

2021-02-27T16:10:00.379+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello getsockname says I am 172.21.15.131:57572 when talking to v2:172.21.15.131:6801/7
2021-02-27T16:10:00.380+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f53dc81c00 0x55f53c0e9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello getsockname says I am 172.21.15.131:57576 when talking to v2:172.21.15.131:6801/7
2021-02-27T16:10:00.381+0000 7f4d420cb700  5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello getsockname says I am 172.21.15.131:57580 when talking to v2:172.21.15.131:6801/7

ever 4 ports

vs

2021-02-27T16:10:00.379+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6bf7a400 msgr2=0x559f60f4b400 unknown :-1 s=STATE_NONE l=0).accept sd=95 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57572/0
2021-02-27T16:10:00.379+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 msgr2=0x559f56e11200 unknown :-1 s=STATE_NONE l=0).accept sd=72 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57574/0
2021-02-27T16:10:00.380+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6cf43800 msgr2=0x559f6c632000 unknown :-1 s=STATE_NONE l=0).accept sd=95 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57576/0
2021-02-27T16:10:00.380+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 msgr2=0x559f56e11200 unknown :-1 s=STATE_NONE l=0).accept sd=72 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57578/0
2021-02-27T16:10:00.381+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6ce7b800 msgr2=0x559f6260a800 unknown :-1 s=STATE_NONE l=0).accept sd=95 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57580/0
2021-02-27T16:10:00.382+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >>  conn(0x559f6b930c00 msgr2=0x559f56e11200 unknown :-1 s=STATE_NONE l=0).accept sd=72 listen_addr v2:172.21.15.131:6801/7 peer_addr v2:172.21.15.131:57582/0

every 2 ports. another process on the same host claiming to be v2:172.21.15.131:6801/7 ?

Actions
Copy link
 #3

Updated by Sage Weil about 3 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 39739
Actions
Copy link
 #4

Updated by Sage Weil about 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link
 #5

Updated by Radoslaw Zarzynski about 1 year ago

  • Related to Bug #57977: osd:tick checking mon for new map added
Actions
Copy link

Also available in: Atom PDF