Project

General

Profile

Actions
Copy link

Bug #40521

closed

cli timeout (e.g., ceph pg dump)

Added by Sage Weil almost 5 years ago. Updated about 2 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

client gets kicked from first mon. reconnects to a second one, but doesn't complete the command.

...

remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700  1 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] <== client.? v1:172.21.15.72:0/3507671956 2 ==== auth(proto 2 36 bytes epoch 0) v1 ==== 66+0+0 (unknown 3351132596 0 0) 0x5603c0ecc480 con 0x5603c1f9c400
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader).paxosservice(auth 1..6) dispatch 0x5603c0ecc480 auth(proto 2 36 bytes epoch 0) v1 from client.? v1:172.21.15.72:0/3507671956 con 0x5603c1f9c400
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader).auth v6 preprocess_query auth(proto 2 36 bytes epoch 0) v1 from client.? v1:172.21.15.72:0/3507671956
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader) e1 ms_handle_authentication session 0x5603c161fa40 con 0x5603c1f9c400 addr v1:172.21.15.72:0/3507671956 MonSession(client.? v1:172.21.15.72:0/3507671956 is open , features 0x3f01cfb8ffadffff (luminous))
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700  1 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] --> v1:172.21.15.72:0/3507671956 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5603c0892480 con 0x5603c1f9c400
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 15 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).send_message inline write is denied, reschedule m=0x5603c0892480
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700  1 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] --> v1:172.21.15.72:0/3507671956 -- mon_map magic: 0 v1 -- 0x5603c13f3200 con 0x5603c1f9c400
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 10 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 15 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).send_message inline write is denied, reschedule m=0x5603c13f3200
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 10 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_event
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).prepare_send_message m auth_reply(proto 2 0 (0) Success) v1
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).prepare_send_message encoding features 4540138292836696063 0x5603c0892480 auth_reply(proto 2 0 (0) Success) v1
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message no session security
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message sending message type=18 src mon.0 front=390 data=0 off 0
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message sending 2 0x5603c0892480
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 10 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 465 remaining bytes 0
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 10 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message sending 0x5603c0892480 done.
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).prepare_send_message m mon_map magic: 0 v1
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).prepare_send_message encoding features 4540138292836696063 0x5603c13f3200 mon_map magic: 0 v1
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message no session security
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message sending message type=4 src mon.0 front=383 data=0 off 0
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message sending 3 0x5603c13f3200
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  0 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  1 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  1 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_message error sending 0x5603c13f3200, (32) Broken pipe
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  1 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_event send msg failed
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  1 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).write_event send msg failed
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).fault
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  1 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).stop
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700  2 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).stop
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 10 --1- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 0x5603c1b62000 :6789 s=OPENED pgs=1 cs=1 l=1).discard_out_queue started
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CLOSED l=1).process
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader) e1 ms_handle_reset 0x5603c1f9c400 v1:172.21.15.72:0/3507671956
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader) e1 reset/close on session client.? v1:172.21.15.72:0/3507671956
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294671a700 20 -- [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5603c1f9c400 legacy=0x5603c1b62000 unknown :6789 s=STATE_CLOSED l=1).process socket closed
remote/smithi072/log/ceph-mon.a.log:2019-06-22T11:51:16.733+0000 7f294871e700 10 mon.a@0(leader) e1 remove_session 0x5603c161fa40 client.? v1:172.21.15.72:0/3507671956 features 0x3f01cfb8ffadffff

then on mon.b,
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >>  conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is v1:172.21.15.72:0/3507671956
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).wait_connect_message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=33
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_connect_message_1 r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept got peer connect_seq 0 global_seq 2
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_message_2 accept of host_type 8, policy.lossy=1 policy.server=1 policy.standby=0 policy.resetcheck=0 features 0x3f01cfb8ffadffff
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_message_2 accept my proto 15, their proto 15
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_message_2 authorizor_protocol 0 len 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_message_2 accept setting up session_security.
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.764+0000 7f779cd5c700 10 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).inject_delay sleep for 0.002
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_message_2 accept new session
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).open
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=2 cs=1 l=1).open accept success, connect_seq = 1 in_seq=0, sending READY
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=2 cs=1 l=1).open accept features 4540138292836696063 authorizer_protocol 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 10 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] accept_conn 0x5643c2f34800 v1:172.21.15.72:0/3507671956
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.766+0000 7f779cd5c700 10 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).inject_delay sleep for 0.002
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 10 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 34 remaining bytes 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=2 cs=1 l=1).handle_ready_connect_message_reply_write r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).wait_seq
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).handle_seq r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700  2 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).handle_seq accept get newly_acked_seq 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).discard_requeued_up_to 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).server_ready session_security is 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=ACCEPTING_HANDLED_CONNECT_MSG pgs=2 cs=1 l=1).server_ready accept done
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).wait_message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message process tag 14
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_keepalive2 r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).append_keepalive_or_ack
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_keepalive2 got KEEPALIVE2 2019-06-22T11:51:16.734555+0000
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).wait_message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message process tag 7
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message begin MSG
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message_header r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message_header got MSG header
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message_header got envelope type=17 src client.? front=60 data=0 off 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=THROTTLE_MESSAGE pgs=2 cs=1 l=1).throttle_message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=THROTTLE_BYTES pgs=2 cs=1 l=1).throttle_bytes
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=THROTTLE_BYTES pgs=2 cs=1 l=1).throttle_bytes wants 60 bytes from policy throttler 0/104857600
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=THROTTLE_DISPATCH_QUEUE pgs=2 cs=1 l=1).throttle_dispatch_queue
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).read_message_front
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=60
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).handle_message_front r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.768+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).handle_message_front got front 60
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).read_message_middle
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).read_message_data_prepare
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).read_message_data msg_left=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_MESSAGE_FRONT pgs=2 cs=1 l=1).read_message_footer
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_FOOTER_AND_DISPATCH pgs=2 cs=1 l=1).handle_message_footer r=0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_FOOTER_AND_DISPATCH pgs=2 cs=1 l=1).handle_message_footer aborted = 0
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_FOOTER_AND_DISPATCH pgs=2 cs=1 l=1).handle_message_footer got 60 + 0 + 0 byte message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_FOOTER_AND_DISPATCH pgs=2 cs=1 l=1).handle_message_footer no session security set
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  5 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=READ_FOOTER_AND_DISPATCH pgs=2 cs=1 l=1). rx client.? seq 1 0x5643c2d9db00 auth(proto 0 30 bytes epoch 0) v1
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).wait_message
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700  1 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] <== client.? v1:172.21.15.72:0/3507671956 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (unknown 673663173 0 0) 0x5643c2d9db00 con 0x5643c2f34800
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  1 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 43
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  1 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message r=-1
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon) e1 _ms_dispatch new session 0x5643c24b2380 MonSession(client.? v1:172.21.15.72:0/3507671956 is open , features 0x3f01cfb8ffadffff (luminous)) features 0x3f01cfb8ffadffff
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  1 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).fault
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon).paxosservice(auth 1..6) dispatch 0x5643c2d9db00 auth(proto 0 30 bytes epoch 0) v1 from client.? v1:172.21.15.72:0/3507671956 con 0x5643c2f34800
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  1 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 20 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).stop
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon).auth v6 preprocess_query auth(proto 0 30 bytes epoch 0) v1 from client.? v1:172.21.15.72:0/3507671956
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700  2 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).stop
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).discard_out_queue started
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 10 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 legacy=0x5643c3683800 unknown :6789 s=STATE_CLOSED l=1).handle_write
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=CLOSED pgs=2 cs=1 l=1).write_event
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700  1 -- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] --> v1:172.21.15.72:0/3507671956 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5643c2834240 con 0x5643c2f34800
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=CLOSED pgs=2 cs=1 l=1).send_message connection closed. Drop message 0x5643c2834240
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon) e1 ms_handle_reset 0x5643c2f34800 v1:172.21.15.72:0/3507671956
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon) e1 reset/close on session client.? v1:172.21.15.72:0/3507671956
remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779f561700 10 mon.b@1(peon) e1 remove_session 0x5643c24b2380 client.? v1:172.21.15.72:0/3507671956 features 0x3f01cfb8ffadffff

nothing from `zgrep 172.21.15.72:0/3507671956 remote/*/log/*client*`.

/a/sage-2019-06-21_16:36:51-rados:thrash-wip-sage2-testing-2019-06-20-2154-distro-basic-smithi/4057742

Related issues 1 (0 open — 1 closed)

Related to RADOS - Bug #39039: mon connection reset, command not resentDuplicateSage Weil

Actions
Actions
Copy link
 #1

Updated by Greg Farnum almost 5 years ago

In the second log snippet there:

remote/smithi203/log/ceph-mon.b.log.gz:2019-06-22T11:51:16.769+0000 7f779cd5c700 1 --1- [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] >> v1:172.21.15.72:0/3507671956 conn(0x5643c2f34800 0x5643c3683800 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed

So did this run just have failure injection turned up and the CLI got unlucky about getting nuked on two separate monitors?

Actions
Copy link
 #2

Updated by Greg Farnum over 4 years ago

  • Related to Bug #39039: mon connection reset, command not resent added
Actions
Copy link
 #3

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link
 #4

Updated by Neha Ojha about 2 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF