Project

General

Profile

Actions
Copy link

Bug #11832

closed

ceph_test_msgr hung in rados suite

Added by David Zafman almost 9 years ago. Updated over 8 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Haomai Wang
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@teuthology:/a/dzafman-2015-05-27_12:16:53-rados-wip-10794---basic-multi/911969

The end of log keeps repeating the same output for days before someone probably killed it.

2015-05-27T17:11:24.619 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 6
2015-05-27T17:11:24.658 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.658584 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x29825000 sd=-1 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=127 cs=33 l=0).fault with nothing to send, going to standby
2015-05-27T17:11:24.658 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.658793 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_IDENTIFY_PEER pgs=175 cs=34 l=0)._try_send injecting socket failure
2015-05-27T17:11:24.659 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.658971 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> :/0 conn(0x2f845000 sd=-1 :16800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed, state=STATE_ACCEPTING_WAIT_BANNER_ADDR
2015-05-27T17:11:24.759 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.759262 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x2f840000 sd=87 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 34 vs existing csq=33 state=STATE_STANDBY
2015-05-27T17:11:24.859 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.859556 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=175 cs=34 l=0).read_until injecting socket failure
2015-05-27T17:11:24.859 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.859761 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x29825000 sd=-1 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=127 cs=33 l=0).fault with nothing to send, going to standby
2015-05-27T17:11:24.860 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.859982 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=175 cs=34 l=0)._try_send injecting socket failure
2015-05-27T17:11:24.860 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.860086 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_BANNER pgs=175 cs=34 l=0).read_until injecting socket failure
2015-05-27T17:11:24.860 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.860193 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x2f853000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=0 cs=0 l=0).read_until injecting socket failure
2015-05-27T17:11:24.860 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.860251 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x2f853000 sd=-1 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed, state=STATE_ACCEPTING_WAIT_CONNECT_MSG
2015-05-27T17:11:24.860 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.860324 7fabce395700  0 -- 127.0.0.1:16800/25805 >> :/0 conn(0x2f84e000 sd=-1 :16800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed, state=STATE_ACCEPTING_WAIT_BANNER_ADDR
2015-05-27T17:11:24.960 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:24.960490 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64e9f000 sd=88 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 34 vs existing csq=33 state=STATE_STANDBY
2015-05-27T17:11:25.060 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.060748 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=175 cs=34 l=0)._try_send injecting socket failure
2015-05-27T17:11:25.261 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.261141 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x29825000 sd=-1 :-1 s=STATE_ACCEPTING_WAIT_SEQ pgs=135 cs=35 l=0).fault with nothing to send, going to standby
2015-05-27T17:11:25.361 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.361496 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64e9a000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 34 vs existing csq=35 state=STATE_STANDBY
2015-05-27T17:11:25.461 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.461881 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64e9a000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 36 vs existing csq=35 state=STATE_STANDBY
2015-05-27T17:11:25.562 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.562186 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=175 cs=36 l=0)._try_send injecting socket failure
2015-05-27T17:11:25.762 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.762632 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x29825000 sd=-1 :-1 s=STATE_ACCEPTING_WAIT_SEQ pgs=137 cs=37 l=0).fault with nothing to send, going to standby
2015-05-27T17:11:25.862 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.862996 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64ea9000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 36 vs existing csq=37 state=STATE_STANDBY
2015-05-27T17:11:25.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.863042 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64ea9000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=0 cs=0 l=0).read_until injecting socket failure
2015-05-27T17:11:25.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.863094 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64ea9000 sd=-1 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed, state=STATE_ACCEPTING_WAIT_CONNECT_MSG
2015-05-27T17:11:25.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.863475 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_CONNECTING_WAIT_BANNER pgs=175 cs=38 l=0)._try_send injecting socket failure
2015-05-27T17:11:25.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.863680 7fabce395700  0 -- 127.0.0.1:16800/25805 >> :/0 conn(0x64ea4000 sd=-1 :16800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed, state=STATE_ACCEPTING_WAIT_BANNER_ADDR
2015-05-27T17:11:25.964 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:25.964075 7fabceb96700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64ebb000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 38 vs existing csq=37 state=STATE_STANDBY
2015-05-27T17:11:26.266 INFO:teuthology.orchestra.run.plana46.stderr:ms_fast_dispatch reply=0 i=661
2015-05-27T17:11:26.266 INFO:teuthology.orchestra.run.plana46.stderr:ms_fast_dispatch reply=0 i=705
2015-05-27T17:11:26.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:26.863184 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=50 :16815 s=STATE_OPEN pgs=178 cs=39 l=0).read_until injecting socket failure
2015-05-27T17:11:26.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:26.863268 7fabceb96700  0 -- 127.0.0.1:16815/25820 >> 127.0.0.1:16800/25805 conn(0x2165b000 sd=-1 :16815 s=STATE_OPEN pgs=178 cs=39 l=0).fault initiating reconnect
2015-05-27T17:11:26.863 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:26.863376 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x29825000 sd=-1 :-1 s=STATE_OPEN pgs=141 cs=39 l=0).fault with nothing to send, going to standby
2015-05-27T17:11:26.963 INFO:teuthology.orchestra.run.plana46.stderr:2015-05-27 17:11:26.963652 7fabce395700  0 -- 127.0.0.1:16800/25805 >> 127.0.0.1:16815/25820 conn(0x64eb6000 sd=52 :16800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 40 vs existing csq=39 state=STATE_STANDBY
2015-05-27T17:11:29.627 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
2015-05-27T17:11:34.634 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
2015-05-27T17:11:39.641 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
.....
2015-05-29T05:01:19.689 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
2015-05-29T05:01:24.696 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
2015-05-29T05:01:29.703 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
2015-05-29T05:01:34.710 INFO:teuthology.orchestra.run.plana46.stderr:available_connections: 60 inflight messages: 4
Actions
Copy link
 #1

Updated by Haomai Wang almost 9 years ago

  • Assignee set to Haomai Wang
Actions
Copy link
 #2

Updated by Haomai Wang over 8 years ago

  • Status changed from New to Duplicate
Actions
Copy link

Also available in: Atom PDF