Project

General

Profile

Actions
Copy link

Bug #10383

closed

unittest_msgr : illegal instruction

Added by Loïc Dachary over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Haomai Wang
Category:
-
Target version:
-
% Done:

100%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

in http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-tarball-trusty-i386-basic/log.cgi?log=5439d9ee708f191a8e1aa758b8edd591be98d806
unittest_msgr fails with illegal instruction (i386). Apparently introduced by http://workbench.dachary.org/ceph/ceph/commit/5439d9ee708f191a8e1aa758b8edd591be98d806

Actions
Copy link
 #1

Updated by Loïc Dachary over 9 years ago 

ubuntu@gitbuilder-ceph-tarball-precise-i386-basic:~/ceph/src$ ./unittest_msgr
[==========] Running 12 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 12 tests from Messenger/MessengerTest
[ RUN      ] Messenger/MessengerTest.SimpleTest/0
SetUp start set up async
ms_handle_fast_connect0xa6955a8
ms_handle_fast_connect con: 0xa6955a8 count: 0
ms_handle_fast_connect0xa6955a8
ms_handle_fast_connect0xa6993b0
ms_handle_fast_connect con: 0xa6993b0 count: 0
ms_fast_dispatch conn: 0xb3b00468 session 0xb3c00690 count: 1
ms_fast_dispatch conn: 0xa6993b0 session 0xb3d00478 count: 1
ms_handle_reset0xa6993b0
ms_handle_reset0xb3b00468
ms_handle_fast_connect0xa691c40
ms_handle_fast_connect con: 0xa691c40 count: 0
ms_handle_fast_connect0xa691c40
ms_handle_fast_connect0xa6955a8
ms_handle_fast_connect0xa69d268
ms_handle_fast_connect con: 0xa69d268 count: 0
ms_fast_dispatch conn: 0xb3b03c08 session 0xb3c00778 count: 1
ms_fast_dispatch conn: 0xa69d268 session 0xb3d00c48 count: 1
ms_handle_reset0xb3b03c08
ms_handle_reset0xb3b03c08
ms_fast_dispatch conn: 0xa6955a8 session 0xa699238 count: 1
ms_handle_reset0xa69d268
ms_handle_reset0xa6993b0
[       OK ] Messenger/MessengerTest.SimpleTest/0 (14 ms)
[ RUN      ] Messenger/MessengerTest.SimpleTest/1
SetUp start set up simple
ms_handle_fast_connect0xa6a1d30
ms_handle_fast_connect con: 0xa6a1d30 count: 0
ms_handle_fast_connect0xa6a1d30
ms_handle_fast_connect0xa6a4818
ms_handle_fast_connect con: 0xa6a4818 count: 0
ms_fast_dispatch conn: 0xb0802660 session 0xb05007e0 count: 1
ms_fast_dispatch conn: 0xa6a4818 session 0xb03007a0 count: 1
ms_handle_reset0xb0802660
2014-12-19 10:04:46.495765 b4efeb40 -1 net_handler: generic_connect connect: %s Connection refused
ms_handle_reset0xa6a4818
ms_handle_fast_connect0xa695310
ms_handle_fast_connect con: 0xa695310 count: 0
ms_handle_fast_connect0xa6a4a58
ms_handle_fast_connect con: 0xa6a4a58 count: 0
ms_fast_dispatch conn: 0xb0502e70 session 0xb0300a28 count: 1
ms_fast_dispatch conn: 0xa6a4a58 session 0xb0400b78 count: 1
ms_handle_reset0xb0502e70
2014-12-19 10:04:46.502190 b7072740  0 -- 127.0.0.1:0/14957 submit_message ping magic: 0 v1 remote, 127.0.0.1:6811/1014957, failed lossy con, dropping message 0xa6a4ed8
ms_fast_dispatch conn: 0xa6a1d30 session 0xa695528 count: 1
[       OK ] Messenger/MessengerTest.SimpleTest/1 (7 ms)
[ RUN      ] Messenger/MessengerTest.NameAddrTest/0
SetUp start set up async
ms_handle_fast_connect0xa6a82b0
ms_handle_fast_connect con: 0xa6a82b0 count: 0
ms_handle_fast_connect0xa6a82b0
ms_handle_fast_connect0xa6aa7c0
ms_handle_fast_connect con: 0xa6aa7c0 count: 0
ms_fast_dispatch conn: 0xb0502f80 session 0xb3c013a8 count: 1
ms_fast_dispatch conn: 0xa6aa7c0 session 0xb3d00f88 count: 1
ms_handle_reset0xb0502f80
ms_handle_reset0xa6aa7c0
ms_handle_reset0xa6aa7c0
[       OK ] Messenger/MessengerTest.NameAddrTest/0 (2 ms)
[ RUN      ] Messenger/MessengerTest.NameAddrTest/1
SetUp start set up simple
ms_handle_fast_connect0xa6ae018
ms_handle_fast_connect con: 0xa6ae018 count: 0
ms_handle_fast_connect0xa6ae018
ms_handle_fast_connect0xa6b09a0
ms_handle_fast_connect con: 0xa6b09a0 count: 0
ms_fast_dispatch conn: 0xb0501e30 session 0xb0300f30 count: 1
ms_fast_dispatch conn: 0xa6b09a0 session 0xb0400c58 count: 1
ms_handle_reset0xb0501e30
ms_handle_reset0xa6b09a0
[       OK ] Messenger/MessengerTest.NameAddrTest/1 (3 ms)
[ RUN      ] Messenger/MessengerTest.FeatureTest/0
SetUp start set up async
ms_handle_fast_connect0xa6b43c8
ms_handle_fast_connect con: 0xa6b43c8 count: 0
ms_handle_reset0xa6b68d8
ms_handle_fast_connect0xa6b43c8
ms_handle_fast_connect con: 0xa6b43c8 count: 0
2014-12-19 10:04:46.509646 b46fdb40  0 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault server, going to standby
ms_handle_fast_connect0xa6b43c8
ms_handle_fast_connect0xa6b8de8
ms_handle_fast_connect con: 0xa6b8de8 count: 0
ms_fast_dispatch conn: 0xb05089a8 session 0xb3c01e78 count: 1
ms_fast_dispatch conn: 0xa6b8de8 session 0xb3d019c8 count: 1
ms_handle_reset0xb0505490
ms_handle_reset0xa6b8de8
ms_handle_reset0xb05089a8
ms_handle_reset0xa6b8de8
[       OK ] Messenger/MessengerTest.FeatureTest/0 (3 ms)
[ RUN      ] Messenger/MessengerTest.FeatureTest/1
SetUp start set up simple
ms_handle_fast_connect0xa6bd330
ms_handle_fast_connect con: 0xa6bd330 count: 0
ms_handle_reset0xa6bfc88
2014-12-19 10:04:46.512646 b07ffb40  0 -- 127.0.0.1:6803/14957 >> :/0 pipe(0xb03015f8 sd=15 :6803 s=0 pgs=0 cs=0 l=0 c=0xb03011d0).accept failed to getpeername (107) Transport endpoint is not connected
ms_handle_reset0xb03011d0
ms_handle_fast_connect0xa6bd330
ms_handle_fast_connect con: 0xa6bd330 count: 0
ms_handle_fast_connect0xa6bd330
ms_handle_fast_connect0xa6bfb60
ms_handle_fast_connect con: 0xa6bfb60 count: 0
ms_fast_dispatch conn: 0xb03011d0 session 0xb0502860 count: 1
ms_fast_dispatch conn: 0xa6bfb60 session 0xb0802d48 count: 1
ms_handle_reset0xb03011d0
[       OK ] Messenger/MessengerTest.FeatureTest/1 (5 ms)
[ RUN      ] Messenger/MessengerTest.StatefulTest/0
SetUp start set up async
ms_handle_fast_connect0xa6c3618
ms_handle_fast_connect con: 0xa6c3618 count: 0
ms_handle_fast_connect0xa6c3618
ms_handle_fast_connect0xa6c5b28
ms_handle_fast_connect con: 0xa6c5b28 count: 0
ms_fast_dispatch conn: 0xb03015f8 session 0xb3c02320 count: 1
ms_fast_dispatch conn: 0xa6c5b28 session 0xb3d020d0 count: 1
2014-12-19 10:04:46.518703 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=15 :6803 s=STATE_OPEN pgs=1 cs=1 l=0).fault server, going to standby
ms_handle_fast_connect0xa6c3618
2014-12-19 10:04:46.519031 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb0304b10 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept connect_seq 0 vs existing 1 state 30
ms_handle_remote_reset0xb03015f8
ms_handle_reset0xb0304b10
2014-12-19 10:04:46.519082 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb0304b10 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept peer reset, then tried to connect to us, replacing
ms_handle_fast_connect0xa6c9040
ms_handle_fast_connect con: 0xa6c9040 count: 0
ms_fast_dispatch conn: 0xb03015f8 session 0xb3c01e78 count: 1
ms_fast_dispatch conn: 0xa6c9040 session 0xb3d024a8 count: 1
2014-12-19 10:04:46.519823 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=16 :6803 s=STATE_OPEN pgs=2 cs=2 l=0).process bad tag 0
2014-12-19 10:04:46.519864 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=16 :6803 s=STATE_OPEN pgs=2 cs=2 l=0).fault with nothing to send, going to standby
2014-12-19 10:04:46.519909 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=2 cs=2 l=0).fault initiating reconnect
ms_handle_fast_connect0xa6c3618
2014-12-19 10:04:46.520275 b4efeb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept we reset (peer sent cseq 3), sending RESETSESSION
ms_handle_remote_reset0xa6c90402014-12-19 10:04:46.520329 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=2 cs=3 l=0).handle_connect_replyconnect got RESETSESSION

ms_handle_fast_connect0xa6c9040
ms_handle_fast_connect con: 0xa6c9040 count: 0
ms_fast_dispatch conn: 0xb03081e0 session 0xb3d030a0 count: 1
ms_fast_dispatch conn: 0xa6c9040 session 0xb3d02d00 count: 1
2014-12-19 10:04:46.520978 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=3 cs=1 l=0).process bad tag 0
2014-12-19 10:04:46.521015 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=3 cs=1 l=0).fault initiating reconnect
2014-12-19 10:04:46.521113 b4efeb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_OPEN pgs=3 cs=1 l=0).fault server, going to standby
ms_handle_fast_connect0xa6c3618
2014-12-19 10:04:46.521514 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb030b970 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept connect_seq 2 vs existing 1 state 30
ms_handle_reset0xb030b970
*** Caught signal (Segmentation fault) **
 in thread b7072740
msg/async/AsyncConnection.cc: In function 'int AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)' thread b4efeb40 time 2014-12-19 10:04:46.521669
msg/async/AsyncConnection.cc: 1386: FAILED assert(reply.connect_seq > connect_seq)
 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xa4) [0x8a9cf05]
 2: (AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)+0x854) [0x8c7b384]
 3: (AsyncConnection::_process_connection()+0x2bd9) [0x8c778b1]
 4: (AsyncConnection::process()+0x4722) [0x8c7437c]
 5: (C_handle_read::do_request(int)+0x28) [0x8c84f0c]
 6: (EventCenter::process_events(int)+0x551) [0x8c1d497]
 7: (Worker::entry()+0x212) [0x8be0c22]
 8: (Thread::entry_wrapper()+0x86) [0x8a99932]
 9: (Thread::_entry_func(void*)+0x1d) [0x8a9989f]
 10: (()+0x6d4c) [0xb7784d4c]
 11: (clone()+0x5e) [0xb7163bae]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-19 10:04:46.529581 b4efeb40 -1 msg/async/AsyncConnection.cc: In function 'int AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)' thread b4efeb40 time 2014-12-19 10:04:46.521669
msg/async/AsyncConnection.cc: 1386: FAILED assert(reply.connect_seq > connect_seq)

 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xa4) [0x8a9cf05]
 2: (AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)+0x854) [0x8c7b384]
 3: (AsyncConnection::_process_connection()+0x2bd9) [0x8c778b1]
 4: (AsyncConnection::process()+0x4722) [0x8c7437c]
 5: (C_handle_read::do_request(int)+0x28) [0x8c84f0c]
 6: (EventCenter::process_events(int)+0x551) [0x8c1d497]
 7: (Worker::entry()+0x212) [0x8be0c22]
 8: (Thread::entry_wrapper()+0x86) [0x8a99932]
 9: (Thread::_entry_func(void*)+0x1d) [0x8a9989f]
 10: (()+0x6d4c) [0xb7784d4c]
 11: (clone()+0x5e) [0xb7163bae]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -309> 2014-12-19 10:04:46.474444 b7072740  5 asok(0xa63bd60) register_command perfcounters_dump hook 0xa63e120
  -308> 2014-12-19 10:04:46.475576 b7072740  5 asok(0xa63bd60) register_command 1 hook 0xa63e120
  -307> 2014-12-19 10:04:46.475584 b7072740  5 asok(0xa63bd60) register_command perf dump hook 0xa63e120
  -306> 2014-12-19 10:04:46.475607 b7072740  5 asok(0xa63bd60) register_command perfcounters_schema hook 0xa63e120
  -305> 2014-12-19 10:04:46.475611 b7072740  5 asok(0xa63bd60) register_command 2 hook 0xa63e120
  -304> 2014-12-19 10:04:46.475615 b7072740  5 asok(0xa63bd60) register_command perf schema hook 0xa63e120
  -303> 2014-12-19 10:04:46.475631 b7072740  5 asok(0xa63bd60) register_command perf reset hook 0xa63e120
  -302> 2014-12-19 10:04:46.475635 b7072740  5 asok(0xa63bd60) register_command config show hook 0xa63e120
  -301> 2014-12-19 10:04:46.475639 b7072740  5 asok(0xa63bd60) register_command config set hook 0xa63e120
  -300> 2014-12-19 10:04:46.475659 b7072740  5 asok(0xa63bd60) register_command config get hook 0xa63e120
  -299> 2014-12-19 10:04:46.475664 b7072740  5 asok(0xa63bd60) register_command config diff hook 0xa63e120
  -298> 2014-12-19 10:04:46.475687 b7072740  5 asok(0xa63bd60) register_command log flush hook 0xa63e120
  -297> 2014-12-19 10:04:46.475692 b7072740  5 asok(0xa63bd60) register_command log dump hook 0xa63e120
  -296> 2014-12-19 10:04:46.475696 b7072740  5 asok(0xa63bd60) register_command log reopen hook 0xa63e120
  -295> 2014-12-19 10:04:46.481567 b7072740  5 asok(0xa63bd60) init /home/ubuntu/ceph/src//test_out/client.admin.14957.asok
  -294> 2014-12-19 10:04:46.481608 b7072740  5 asok(0xa63bd60) bind_and_listen /home/ubuntu/ceph/src//test_out/client.admin.14957.asok
  -293> 2014-12-19 10:04:46.481862 b7072740  5 asok(0xa63bd60) register_command 0 hook 0xa63ff48
  -292> 2014-12-19 10:04:46.481919 b7072740  5 asok(0xa63bd60) register_command version hook 0xa63ff48
  -291> 2014-12-19 10:04:46.481925 b7072740  5 asok(0xa63bd60) register_command git_version hook 0xa63ff48
  -290> 2014-12-19 10:04:46.481930 b7072740  5 asok(0xa63bd60) register_command help hook 0xa640390
  -289> 2014-12-19 10:04:46.481934 b7072740  5 asok(0xa63bd60) register_command get_command_descriptions hook 0xa640498
  -288> 2014-12-19 10:04:46.482008 b606bb40  5 asok(0xa63bd60) entry start
  -287> 2014-12-19 10:04:46.491456 b7072740  1 -- 127.0.0.1:0/0 learned_addr learned my addr 127.0.0.1:0/0
  -286> 2014-12-19 10:04:46.491553 b7072740  1  Processor -- bind bind my_inst.addr is 127.0.0.1:6800/14957
  -285> 2014-12-19 10:04:46.491560 b7072740  1  Processor -- start start
  -284> 2014-12-19 10:04:46.491651 b7072740  1 -- 127.0.0.1:6800/14957 start start
  -283> 2014-12-19 10:04:46.491779 b7072740  1  Processor -- start start
  -282> 2014-12-19 10:04:46.491826 b7072740  1 -- :/0 start start
  -281> 2014-12-19 10:04:46.491960 b7072740  1 Event wakeup
  -280> 2014-12-19 10:04:46.492198 b56ffb40  1 Event wakeup
  -279> 2014-12-19 10:04:46.492302 b46fdb40  1 -- 127.0.0.1:6800/14957 >> :/0 conn(0xb3b00468 sd=15 :6800 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=15 127.0.0.1:36161/0
  -278> 2014-12-19 10:04:46.492406 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
  -277> 2014-12-19 10:04:46.492573 b46fdb40  1 Event wakeup
  -276> 2014-12-19 10:04:46.492640 b4efeb40  1 Event wakeup
  -275> 2014-12-19 10:04:46.492754 b4efeb40  1 Event wakeup
  -274> 2014-12-19 10:04:46.492824 b46fdb40  2 -- 127.0.0.1:6800/14957 >> 127.0.0.1:0/14957 conn(0xb3b00468 sd=15 :6800 s=STATE_ACCEPTING_READY pgs=1 cs=1 l=1).handle_connect_msg accept write reply msg done
  -273> 2014-12-19 10:04:46.493092 b7072740  1 -- 127.0.0.1:6800/14957 rebind rebind avoid 6800,6801,6802,6803,6804,6805,6806,6807,6808,6809
  -272> 2014-12-19 10:04:46.493299 b7072740  1 -- 127.0.0.1:6800/14957 mark_down_all 
  -271> 2014-12-19 10:04:46.493326 b7072740  5 -- 127.0.0.1:6800/14957 mark_down_all 127.0.0.1:0/14957 0xb3b00468
  -270> 2014-12-19 10:04:46.493333 b7072740  1 Event wakeup
  -269> 2014-12-19 10:04:46.493386 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6800/14957 conn(0xa6993b0 sd=14 :0 s=STATE_OPEN pgs=1 cs=1 l=1).read_bulk Peer close file descriptor 14
  -268> 2014-12-19 10:04:46.493404 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6800/14957 conn(0xa6993b0 sd=14 :0 s=STATE_OPEN pgs=1 cs=1 l=1).process read tag failed, state is STATE_OPEN
  -267> 2014-12-19 10:04:46.493411 b4efeb40  1 Event wakeup
  -266> 2014-12-19 10:04:46.493460 b4efeb40  1 Event wakeup
  -265> 2014-12-19 10:04:46.493535 b46fdb40  1 Event wakeup
  -264> 2014-12-19 10:04:46.493700 b7072740  1  Processor -- rebind rebind avoid 6800,6801,6802,6803,6804,6805,6806,6807,6808,6809
  -263> 2014-12-19 10:04:46.493807 b7072740  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
  -262> 2014-12-19 10:04:46.493990 b7072740  1  Processor -- bind bind my_inst.addr is 127.0.0.1:6810/1014957
  -261> 2014-12-19 10:04:46.494040 b7072740  1  Processor -- start start
  -260> 2014-12-19 10:04:46.494106 b7072740  1 Event wakeup
  -259> 2014-12-19 10:04:46.494281 b56ffb40  1 Event wakeup
  -258> 2014-12-19 10:04:46.494314 b46fdb40  1 -- 127.0.0.1:6810/1014957 >> :/0 conn(0xb3b03c08 sd=15 :6810 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=15 127.0.0.1:36575/0
  -257> 2014-12-19 10:04:46.494382 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
  -256> 2014-12-19 10:04:46.494544 b46fdb40  1 Event wakeup
  -255> 2014-12-19 10:04:46.494571 b46fdb40  2 -- 127.0.0.1:6810/1014957 >> 127.0.0.1:0/14957 conn(0xb3b03c08 sd=15 :6810 s=STATE_ACCEPTING_READY pgs=2 cs=1 l=1).handle_connect_msg accept write reply msg done
  -254> 2014-12-19 10:04:46.494606 b4efeb40  1 Event wakeup
  -253> 2014-12-19 10:04:46.494720 b4efeb40  1 Event wakeup
  -252> 2014-12-19 10:04:46.494971 b7072740  1 Event wakeup
  -251> 2014-12-19 10:04:46.495032 b4efeb40  1 Event wakeup
  -250> 2014-12-19 10:04:46.495088 b7072740  1 -- 127.0.0.1:6810/1014957 mark_down_all 
  -249> 2014-12-19 10:04:46.495087 b46fdb40  1 -- 127.0.0.1:6810/1014957 >> 127.0.0.1:0/14957 conn(0xb3b03c08 sd=15 :6810 s=STATE_OPEN pgs=2 cs=1 l=1).read_bulk Peer close file descriptor 15
  -248> 2014-12-19 10:04:46.495114 b7072740  5 -- 127.0.0.1:6810/1014957 mark_down_all 127.0.0.1:0/14957 0xb3b03c08
  -247> 2014-12-19 10:04:46.495159 b46fdb40  1 -- 127.0.0.1:6810/1014957 >> 127.0.0.1:0/14957 conn(0xb3b03c08 sd=15 :6810 s=STATE_OPEN pgs=2 cs=1 l=1).process read tag failed, state is STATE_OPEN
  -246> 2014-12-19 10:04:46.495174 b46fdb40  1 Event wakeup
  -245> 2014-12-19 10:04:46.495176 b7072740  1 Event wakeup
  -244> 2014-12-19 10:04:46.495206 b46fdb40  1 Event wakeup
  -243> 2014-12-19 10:04:46.495271 b46fdb40  1 Event wakeup
  -242> 2014-12-19 10:04:46.495476 b7072740  1 -- 127.0.0.1:6810/1014957 wait complete.
  -241> 2014-12-19 10:04:46.495581 b7072740  1 Event wakeup
  -240> 2014-12-19 10:04:46.495603 b7072740  1 Event wakeup
  -239> 2014-12-19 10:04:46.495765 b4efeb40 -1 net_handler: generic_connect connect: %s Connection refused
  -238> 2014-12-19 10:04:46.495777 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all 
  -237> 2014-12-19 10:04:46.495817 b7072740  5 -- 127.0.0.1:0/14957 mark_down_all 127.0.0.1:6810/1014957 0xa69d268
  -236> 2014-12-19 10:04:46.495835 b7072740  1 Event wakeup
  -235> 2014-12-19 10:04:46.495851 b4efeb40  1 Event wakeup
  -234> 2014-12-19 10:04:46.495947 b7072740  5 -- 127.0.0.1:0/14957 mark_down_all 127.0.0.1:6800/14957 0xa6993b0
  -233> 2014-12-19 10:04:46.495957 b7072740  1 Event wakeup
  -232> 2014-12-19 10:04:46.496017 b4efeb40  1 Event wakeup
  -231> 2014-12-19 10:04:46.496166 b7072740  1 -- 127.0.0.1:0/14957 wait complete.
  -230> 2014-12-19 10:04:46.496397 b7072740  1 -- 127.0.0.1:0/0 learned my addr 127.0.0.1:0/0
  -229> 2014-12-19 10:04:46.496411 b7072740  1 accepter.accepter.bind my_inst.addr is 127.0.0.1:6801/14957 need_addr=0
  -228> 2014-12-19 10:04:46.496583 b7072740  1 accepter.accepter.start
  -227> 2014-12-19 10:04:46.496731 b7072740  1 -- 127.0.0.1:6801/14957 messenger.start
  -226> 2014-12-19 10:04:46.497007 b7072740  1 -- :/0 messenger.start
  -225> 2014-12-19 10:04:46.497362 b7072740  1 -- :/14957 --> 127.0.0.1:6801/14957 -- ping magic: 0 v1 -- ?+0 0xa6a24f8 con 0xa6a4818
  -224> 2014-12-19 10:04:46.497916 b07ffb40  1 -- 127.0.0.1:6801/14957 >> :/0 pipe(0xb0800468 sd=16 :6801 s=0 pgs=0 cs=0 l=0 c=0xb0802660).accept sd=16 127.0.0.1:52217/0
  -223> 2014-12-19 10:04:46.497930 b0af9b40  1 -- 127.0.0.1:0/14957 learned my addr 127.0.0.1:0/14957
  -222> 2014-12-19 10:04:46.498539 b07ffb40  1 -- 127.0.0.1:6801/14957 <== client.? 127.0.0.1:0/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0500990 con 0xb0802660
  -221> 2014-12-19 10:04:46.498571 b07ffb40  1 -- 127.0.0.1:6801/14957 --> 127.0.0.1:0/14957 -- ping magic: 0 v1 -- ?+0 0xb0500ab8 con 0xb0802660
  -220> 2014-12-19 10:04:46.498773 b01feb40  1 -- 127.0.0.1:0/14957 <== osd.0 127.0.0.1:6801/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0400468 con 0xa6a4818
  -219> 2014-12-19 10:04:46.498923 b7072740  1 -- 127.0.0.1:6801/14957 rebind avoid 6801,6802,6803,6804,6805,6806,6807,6808,6809,6810
  -218> 2014-12-19 10:04:46.499030 b7072740  1 -- 127.0.0.1:6801/14957 mark_down_all
  -217> 2014-12-19 10:04:46.499042 b7072740  5 -- 127.0.0.1:6801/14957 mark_down_all 127.0.0.1:0/14957 0xb0800468
  -216> 2014-12-19 10:04:46.499123 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6801/14957 pipe(0xa6a2620 sd=15 :52217 s=2 pgs=1 cs=1 l=1 c=0xa6a4818).reader couldn't read tag, (0) Success
  -215> 2014-12-19 10:04:46.499181 b7072740  1 accepter.accepter.rebind avoid 6801,6802,6803,6804,6805,6806,6807,6808,6809,6810
  -214> 2014-12-19 10:04:46.499205 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6801/14957 pipe(0xa6a2620 sd=15 :52217 s=2 pgs=1 cs=1 l=1 c=0xa6a4818).fault (0) Success
  -213> 2014-12-19 10:04:46.499363 b07ffb40  2 -- 127.0.0.1:6811/1014957 >> 127.0.0.1:0/14957 pipe(0xb0800468 sd=16 :6801 s=4 pgs=1 cs=1 l=1 c=0xb0802660).reader couldn't read tag, (0) Success
  -212> 2014-12-19 10:04:46.499389 b07ffb40  2 -- 127.0.0.1:6811/1014957 >> 127.0.0.1:0/14957 pipe(0xb0800468 sd=16 :6801 s=4 pgs=1 cs=1 l=1 c=0xb0802660).fault (0) Success
  -211> 2014-12-19 10:04:46.499712 b7072740  1 accepter.accepter.bind my_inst.addr is 127.0.0.1:6811/1014957 need_addr=0
  -210> 2014-12-19 10:04:46.499736 b7072740  1 accepter.accepter.start
  -209> 2014-12-19 10:04:46.499824 b7072740  1 -- 127.0.0.1:0/14957 --> 127.0.0.1:6811/1014957 -- ping magic: 0 v1 -- ?+0 0xa6a4c18 con 0xa6a4a58
  -208> 2014-12-19 10:04:46.500152 b02ffb40  1 -- 127.0.0.1:6811/1014957 >> :/0 pipe(0xb0500c78 sd=15 :6811 s=0 pgs=0 cs=0 l=0 c=0xb0502e70).accept sd=15 127.0.0.1:50536/0
  -207> 2014-12-19 10:04:46.500655 b02ffb40  1 -- 127.0.0.1:6811/1014957 <== client.? 127.0.0.1:0/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0300b10 con 0xb0502e70
  -206> 2014-12-19 10:04:46.500856 b02ffb40  1 -- 127.0.0.1:6811/1014957 --> 127.0.0.1:0/14957 -- ping magic: 0 v1 -- ?+0 0xb0300c38 con 0xb0502e70
  -205> 2014-12-19 10:04:46.501103 b0af9b40  1 -- 127.0.0.1:0/14957 <== osd.0 127.0.0.1:6811/1014957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0802770 con 0xa6a4a58
  -204> 2014-12-19 10:04:46.501238 b7072740  1 -- 127.0.0.1:0/14957 mark_down 0xa6a4a58 -- 0xa6a2620
  -203> 2014-12-19 10:04:46.501350 b7072740  1 -- 127.0.0.1:6811/1014957 mark_down_all
  -202> 2014-12-19 10:04:46.501365 b0af9b40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6811/1014957 pipe(0xa6a2620 sd=13 :50536 s=4 pgs=2 cs=1 l=1 c=0xa6a4a58).reader couldn't read tag, (0) Success
  -201> 2014-12-19 10:04:46.501393 b0af9b40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6811/1014957 pipe(0xa6a2620 sd=13 :50536 s=4 pgs=2 cs=1 l=1 c=0xa6a4a58).fault (0) Success
  -200> 2014-12-19 10:04:46.501393 b7072740  5 -- 127.0.0.1:6811/1014957 mark_down_all 127.0.0.1:0/14957 0xb0500c78
  -199> 2014-12-19 10:04:46.501336 b02ffb40  2 -- 127.0.0.1:6811/1014957 >> 127.0.0.1:0/14957 pipe(0xb0500c78 sd=15 :6811 s=2 pgs=2 cs=1 l=1 c=0xb0502e70).reader couldn't read tag, (0) Success
  -198> 2014-12-19 10:04:46.501645 b02ffb40  2 -- 127.0.0.1:6811/1014957 >> 127.0.0.1:0/14957 pipe(0xb0500c78 sd=15 :6811 s=2 pgs=2 cs=1 l=1 c=0xb0502e70).fault (0) Success
  -197> 2014-12-19 10:04:46.502104 b7072740  1 -- 127.0.0.1:6811/1014957 shutdown complete.
  -196> 2014-12-19 10:04:46.502148 b7072740  1 -- 127.0.0.1:0/14957 --> 127.0.0.1:6811/1014957 -- ping magic: 0 v1 -- ?+0 0xa6a4ed8 con 0xa6a4a58
  -195> 2014-12-19 10:04:46.502190 b7072740  0 -- 127.0.0.1:0/14957 submit_message ping magic: 0 v1 remote, 127.0.0.1:6811/1014957, failed lossy con, dropping message 0xa6a4ed8
  -194> 2014-12-19 10:04:46.502269 b7072740  1 -- 127.0.0.1:0/14957 --> 127.0.0.1:0/14957 -- ping magic: 0 v1 -- ?+0 0xa6a4ed8 con 0xa6a1d30
  -193> 2014-12-19 10:04:46.502393 b1afbb40  1 -- 127.0.0.1:0/14957 <== client.? 127.0.0.1:0/14957 0 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xa6a4ed8 con 0xa6a1d30
  -192> 2014-12-19 10:04:46.502510 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all
  -191> 2014-12-19 10:04:46.503041 b7072740  1 -- 127.0.0.1:0/14957 shutdown complete.
  -190> 2014-12-19 10:04:46.503268 b7072740  1 -- 127.0.0.1:0/0 learned_addr learned my addr 127.0.0.1:0/0
  -189> 2014-12-19 10:04:46.503295 b7072740  1  Processor -- bind bind my_inst.addr is 127.0.0.1:6801/14957
  -188> 2014-12-19 10:04:46.503305 b7072740  1  Processor -- start start
  -187> 2014-12-19 10:04:46.503367 b7072740  1 -- 127.0.0.1:6801/14957 start start
  -186> 2014-12-19 10:04:46.503396 b7072740  1  Processor -- start start
  -185> 2014-12-19 10:04:46.503401 b7072740  1 -- :/0 start start
  -184> 2014-12-19 10:04:46.503506 b7072740  1 Event wakeup
  -183> 2014-12-19 10:04:46.503628 b12fab40  1 Event wakeup
  -182> 2014-12-19 10:04:46.503662 b46fdb40  1 -- 127.0.0.1:6801/14957 >> :/0 conn(0xb0502f80 sd=15 :6801 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=15 127.0.0.1:52219/0
  -181> 2014-12-19 10:04:46.503732 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
  -180> 2014-12-19 10:04:46.503881 b46fdb40  1 Event wakeup
  -179> 2014-12-19 10:04:46.503909 b46fdb40  2 -- 127.0.0.1:6801/14957 >> 127.0.0.1:0/14957 conn(0xb0502f80 sd=15 :6801 s=STATE_ACCEPTING_READY pgs=1 cs=1 l=1).handle_connect_msg accept write reply msg done
  -178> 2014-12-19 10:04:46.503941 b4efeb40  1 Event wakeup
  -177> 2014-12-19 10:04:46.504029 b4efeb40  1 Event wakeup
  -176> 2014-12-19 10:04:46.504332 b7072740  1 -- 127.0.0.1:6801/14957 mark_down_all 
  -175> 2014-12-19 10:04:46.504360 b7072740  5 -- 127.0.0.1:6801/14957 mark_down_all 127.0.0.1:0/14957 0xb0502f80
  -174> 2014-12-19 10:04:46.504371 b7072740  1 Event wakeup
  -173> 2014-12-19 10:04:46.504460 b46fdb40  1 Event wakeup
  -172> 2014-12-19 10:04:46.504530 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6801/14957 conn(0xa6aa7c0 sd=14 :0 s=STATE_OPEN pgs=1 cs=1 l=1).read_bulk Peer close file descriptor 14
  -171> 2014-12-19 10:04:46.504549 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6801/14957 conn(0xa6aa7c0 sd=14 :0 s=STATE_OPEN pgs=1 cs=1 l=1).process read tag failed, state is STATE_OPEN
  -170> 2014-12-19 10:04:46.504560 b4efeb40  1 Event wakeup
  -169> 2014-12-19 10:04:46.504598 b4efeb40  1 Event wakeup
  -168> 2014-12-19 10:04:46.504812 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all 
  -167> 2014-12-19 10:04:46.504840 b7072740  5 -- 127.0.0.1:0/14957 mark_down_all 127.0.0.1:6801/14957 0xa6aa7c0
  -166> 2014-12-19 10:04:46.504847 b7072740  1 Event wakeup
  -165> 2014-12-19 10:04:46.504880 b4efeb40  1 Event wakeup
  -164> 2014-12-19 10:04:46.504982 b7072740  1 -- 127.0.0.1:6801/14957 wait complete.
  -163> 2014-12-19 10:04:46.505013 b7072740  1 -- 127.0.0.1:0/14957 wait complete.
  -162> 2014-12-19 10:04:46.505235 b7072740  1 -- 127.0.0.1:0/0 learned my addr 127.0.0.1:0/0
  -161> 2014-12-19 10:04:46.505314 b7072740  1 accepter.accepter.bind my_inst.addr is 127.0.0.1:6802/14957 need_addr=0
  -160> 2014-12-19 10:04:46.505431 b7072740  1 accepter.accepter.start
  -159> 2014-12-19 10:04:46.505513 b7072740  1 -- 127.0.0.1:6802/14957 messenger.start
  -158> 2014-12-19 10:04:46.505786 b7072740  1 -- :/0 messenger.start
  -157> 2014-12-19 10:04:46.506133 b7072740  1 -- :/14957 --> 127.0.0.1:6802/14957 -- ping magic: 0 v1 -- ?+0 0xa6ae680 con 0xa6b09a0
  -156> 2014-12-19 10:04:46.506501 b07ffb40  1 -- 127.0.0.1:6802/14957 >> :/0 pipe(0xb0505490 sd=15 :6802 s=0 pgs=0 cs=0 l=0 c=0xb0501e30).accept sd=15 127.0.0.1:48698/0
  -155> 2014-12-19 10:04:46.506526 b0af9b40  1 -- 127.0.0.1:0/14957 learned my addr 127.0.0.1:0/14957
  -154> 2014-12-19 10:04:46.507290 b07ffb40  1 -- 127.0.0.1:6802/14957 <== client.? 127.0.0.1:0/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0300fa8 con 0xb0501e30
  -153> 2014-12-19 10:04:46.507423 b07ffb40  1 -- 127.0.0.1:6802/14957 --> 127.0.0.1:0/14957 -- ping magic: 0 v1 -- ?+0 0xb03011d0 con 0xb0501e30
  -152> 2014-12-19 10:04:46.507579 b01feb40  1 -- 127.0.0.1:0/14957 <== osd.0 127.0.0.1:6802/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0800468 con 0xa6b09a0
  -151> 2014-12-19 10:04:46.507779 b7072740  1 -- 127.0.0.1:6802/14957 mark_down_all
  -150> 2014-12-19 10:04:46.507813 b7072740  5 -- 127.0.0.1:6802/14957 mark_down_all 127.0.0.1:0/14957 0xb0505490
  -149> 2014-12-19 10:04:46.507889 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6802/14957 pipe(0xa6ae7a8 sd=14 :48698 s=2 pgs=1 cs=1 l=1 c=0xa6b09a0).reader couldn't read tag, (0) Success
  -148> 2014-12-19 10:04:46.507945 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all
  -147> 2014-12-19 10:04:46.507954 b7072740  5 -- 127.0.0.1:0/14957 mark_down_all 127.0.0.1:6802/14957 0xa6ae7a8
  -146> 2014-12-19 10:04:46.507946 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6802/14957 pipe(0xa6ae7a8 sd=14 :48698 s=2 pgs=1 cs=1 l=1 c=0xa6b09a0).fault (0) Success
  -145> 2014-12-19 10:04:46.508020 b07ffb40  2 -- 127.0.0.1:6802/14957 >> 127.0.0.1:0/14957 pipe(0xb0505490 sd=15 :6802 s=4 pgs=1 cs=1 l=1 c=0xb0501e30).reader couldn't read tag, (0) Success
  -144> 2014-12-19 10:04:46.508058 b07ffb40  2 -- 127.0.0.1:6802/14957 >> 127.0.0.1:0/14957 pipe(0xb0505490 sd=15 :6802 s=4 pgs=1 cs=1 l=1 c=0xb0501e30).fault (0) Success
  -143> 2014-12-19 10:04:46.508456 b7072740  1 -- 127.0.0.1:6802/14957 shutdown complete.
  -142> 2014-12-19 10:04:46.508779 b7072740  1 -- 127.0.0.1:0/14957 shutdown complete.
  -141> 2014-12-19 10:04:46.509032 b7072740  1 -- 127.0.0.1:0/0 learned_addr learned my addr 127.0.0.1:0/0
  -140> 2014-12-19 10:04:46.509060 b7072740  1  Processor -- bind bind my_inst.addr is 127.0.0.1:6802/14957
  -139> 2014-12-19 10:04:46.509069 b7072740  1  Processor -- start start
  -138> 2014-12-19 10:04:46.509117 b7072740  1 -- 127.0.0.1:6802/14957 start start
  -137> 2014-12-19 10:04:46.509149 b7072740  1  Processor -- start start
  -136> 2014-12-19 10:04:46.509154 b7072740  1 -- :/0 start start
  -135> 2014-12-19 10:04:46.509307 b7072740  1 Event wakeup
  -134> 2014-12-19 10:04:46.509344 b7072740  1 -- :/14957 mark_down_all 
  -133> 2014-12-19 10:04:46.509351 b7072740  5 -- :/14957 mark_down_all 127.0.0.1:6802/14957 0xa6b68d8
 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::BackTrace::BackTrace(int)+0x30) [0x893d5ba]
 2: ./unittest_msgr() [0x893cbd1]
 3: [0xb77bd400]
 4: (FakeDispatcher::Session::get_count()+0x6) [0x8915eac]
 5: (MessengerTest_StatefulTest_Test::TestBody()+0x16d9) [0x8910779]
 6: (testing::Test::Run()+0x7c) [0x8920b44]
 7: (testing::internal::TestInfoImpl::Run()+0xbb) [0x8921003]
 8: (testing::TestCase::Run()+0xb8) [0x892145c]
 9: (testing::internal::UnitTestImpl::RunAllTests()+0x258) [0x8925202]
 10: (testing::UnitTest::Run()+0x14) [0x8924390]
 11: (main()+0x120) [0x8913c71]
 12: (__libc_start_main()+0xf3) [0xb708d4d3]
 13: ./unittest_msgr() [0x890c221]
  -132> 2014-12-19 10:04:46.509359 b7072740  1 Event wakeup
  -131> 2014-12-19 10:04:46.509420 b4efeb40  1 Event wakeup
  -130> 2014-12-19 10:04:46.509497 b32feb40  1 Event wakeup
  -129> 2014-12-19 10:04:46.509529 b7072740  1 -- :/14957 wait complete.
  -128> 2014-12-19 10:04:46.509553 b7072740  1 -- :/14957 start start
  -127> 2014-12-19 10:04:46.509563 b46fdb40  1 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=14 127.0.0.1:48699/0
  -126> 2014-12-19 10:04:46.509609 b46fdb40  1 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).read_bulk Peer close file descriptor 14
  -125> 2014-12-19 10:04:46.509622 b46fdb40  1 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).read_until read failed, state is STATE_ACCEPTING_WAIT_BANNER_ADDR
  -124> 2014-12-19 10:04:46.509631 b46fdb40  1 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection read peer banner and addr failed
  -123> 2014-12-19 10:04:46.509646 b46fdb40  0 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb0505490 sd=14 :6802 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault server, going to standby
  -122> 2014-12-19 10:04:46.509658 b7072740  1 Event wakeup
  -121> 2014-12-19 10:04:46.509737 b7072740  1 Event wakeup
  -120> 2014-12-19 10:04:46.509835 b32feb40  1 Event wakeup
  -119> 2014-12-19 10:04:46.509868 b46fdb40  1 -- 127.0.0.1:6802/14957 >> :/0 conn(0xb05089a8 sd=16 :6802 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=16 127.0.0.1:48700/0
  -118> 2014-12-19 10:04:46.509984 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
  -117> 2014-12-19 10:04:46.510151 b46fdb40  1 Event wakeup
  -116> 2014-12-19 10:04:46.510204 b46fdb40  2 -- 127.0.0.1:6802/14957 >> 127.0.0.1:0/14957 conn(0xb05089a8 sd=16 :6802 s=STATE_ACCEPTING_READY pgs=2 cs=1 l=1).handle_connect_msg accept write reply msg done
  -115> 2014-12-19 10:04:46.510227 b4efeb40  1 Event wakeup
  -114> 2014-12-19 10:04:46.510324 b4efeb40  1 Event wakeup
  -113> 2014-12-19 10:04:46.510646 b7072740  1 -- 127.0.0.1:6802/14957 mark_down_all 
  -112> 2014-12-19 10:04:46.510669 b7072740  5 -- 127.0.0.1:6802/14957 mark_down_all accepting_conn 0xb0505490
  -111> 2014-12-19 10:04:46.510677 b7072740  1 Event wakeup
  -110> 2014-12-19 10:04:46.510744 b46fdb40  1 Event wakeup
  -109> 2014-12-19 10:04:46.510825 b7072740  5 -- 127.0.0.1:6802/14957 mark_down_all 127.0.0.1:0/14957 0xb05089a8
  -108> 2014-12-19 10:04:46.510850 b7072740  1 Event wakeup
  -107> 2014-12-19 10:04:46.510944 b46fdb40  1 Event wakeup
  -106> 2014-12-19 10:04:46.510950 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6802/14957 conn(0xa6b8de8 sd=15 :0 s=STATE_OPEN pgs=1 cs=1 l=1).read_bulk Peer close file descriptor 15
  -105> 2014-12-19 10:04:46.510964 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6802/14957 conn(0xa6b8de8 sd=15 :0 s=STATE_OPEN pgs=1 cs=1 l=1).process read tag failed, state is STATE_OPEN
  -104> 2014-12-19 10:04:46.510982 b4efeb40  1 Event wakeup
  -103> 2014-12-19 10:04:46.511019 b4efeb40  1 Event wakeup
  -102> 2014-12-19 10:04:46.511331 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all 
  -101> 2014-12-19 10:04:46.511358 b7072740  5 -- 127.0.0.1:0/14957 mark_down_all 127.0.0.1:6802/14957 0xa6b8de8
  -100> 2014-12-19 10:04:46.511369 b7072740  1 Event wakeup
   -99> 2014-12-19 10:04:46.511414 b4efeb40  1 Event wakeup
   -98> 2014-12-19 10:04:46.511526 b7072740  1 -- 127.0.0.1:6802/14957 wait complete.
   -97> 2014-12-19 10:04:46.511548 b7072740  1 -- 127.0.0.1:0/14957 wait complete.
   -96> 2014-12-19 10:04:46.511686 b7072740  1 -- 127.0.0.1:0/0 learned my addr 127.0.0.1:0/0
   -95> 2014-12-19 10:04:46.511712 b7072740  1 accepter.accepter.bind my_inst.addr is 127.0.0.1:6803/14957 need_addr=0
   -94> 2014-12-19 10:04:46.511796 b7072740  1 accepter.accepter.start
   -93> 2014-12-19 10:04:46.511841 b7072740  1 -- 127.0.0.1:6803/14957 messenger.start
   -92> 2014-12-19 10:04:46.512004 b7072740  1 -- :/0 messenger.start
   -91> 2014-12-19 10:04:46.512282 b7072740  1 -- :/14957 --> 127.0.0.1:6803/14957 -- ping magic: 0 v1 -- ?+0 0xa6bd968 con 0xa6bfc88
   -90> 2014-12-19 10:04:46.512347 b7072740  1 -- :/14957 mark_down_all
   -89> 2014-12-19 10:04:46.512360 b7072740  5 -- :/14957 mark_down_all 127.0.0.1:6803/14957 0xa6bda90
   -88> 2014-12-19 10:04:46.512447 b0af9b40  2 -- :/14957 >> 127.0.0.1:6803/14957 pipe(0xa6bda90 sd=14 :0 s=4 pgs=0 cs=0 l=1 c=0xa6bfc88).connect couldn't read banner, (0) Success
   -87> 2014-12-19 10:04:46.512646 b07ffb40  0 -- 127.0.0.1:6803/14957 >> :/0 pipe(0xb03015f8 sd=15 :6803 s=0 pgs=0 cs=0 l=0 c=0xb03011d0).accept failed to getpeername (107) Transport endpoint is not connected
   -86> 2014-12-19 10:04:46.512726 b07ffb40  2 -- 127.0.0.1:6803/14957 >> :/0 pipe(0xb03015f8 sd=15 :6803 s=4 pgs=0 cs=0 l=0 c=0xb03011d0).fault (107) Transport endpoint is not connected
   -85> 2014-12-19 10:04:46.512637 b0af9b40  3 -- :/14957 >> 127.0.0.1:6803/14957 pipe(0xa6bda90 sd=14 :0 s=4 pgs=0 cs=0 l=1 c=0xa6bfc88).connect fault, but state = closed != connecting, stopping
   -84> 2014-12-19 10:04:46.513000 b7072740  1 -- :/14957 shutdown complete.
   -83> 2014-12-19 10:04:46.513037 b7072740  1 -- :/14957 messenger.start
   -82> 2014-12-19 10:04:46.513534 b7072740  1 -- :/14957 --> 127.0.0.1:6803/14957 -- ping magic: 0 v1 -- ?+0 0xa6bfc70 con 0xa6bfb60
   -81> 2014-12-19 10:04:46.513559 b07ffb40  1 -- 127.0.0.1:6803/14957 >> :/0 pipe(0xb03015f8 sd=15 :6803 s=0 pgs=0 cs=0 l=0 c=0xb03011d0).accept sd=15 127.0.0.1:52576/0
   -80> 2014-12-19 10:04:46.513617 b0af9b40  1 -- 127.0.0.1:0/14957 learned my addr 127.0.0.1:0/14957
   -79> 2014-12-19 10:04:46.514073 b07ffb40  1 -- 127.0.0.1:6803/14957 <== client.? 127.0.0.1:0/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0502618 con 0xb03011d0
   -78> 2014-12-19 10:04:46.514180 b07ffb40  1 -- 127.0.0.1:6803/14957 --> 127.0.0.1:0/14957 -- ping magic: 0 v1 -- ?+0 0xb0502a80 con 0xb03011d0
   -77> 2014-12-19 10:04:46.514341 b01feb40  1 -- 127.0.0.1:0/14957 <== osd.0 127.0.0.1:6803/14957 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0xb0400890 con 0xa6bfb60
   -76> 2014-12-19 10:04:46.514483 b7072740  1 -- 127.0.0.1:6803/14957 mark_down_all
   -75> 2014-12-19 10:04:46.514512 b7072740  5 -- 127.0.0.1:6803/14957 mark_down_all 127.0.0.1:0/14957 0xb03015f8
   -74> 2014-12-19 10:04:46.514578 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 pipe(0xa6bd968 sd=14 :52576 s=2 pgs=1 cs=1 l=1 c=0xa6bfb60).reader couldn't read tag, (0) Success
   -73> 2014-12-19 10:04:46.514609 b01feb40  2 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 pipe(0xa6bd968 sd=14 :52576 s=2 pgs=1 cs=1 l=1 c=0xa6bfb60).fault (0) Success
   -72> 2014-12-19 10:04:46.514650 b7072740  1 -- 127.0.0.1:0/14957 mark_down_all
   -71> 2014-12-19 10:04:46.514688 b07ffb40  2 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 pipe(0xb03015f8 sd=15 :6803 s=4 pgs=2 cs=1 l=1 c=0xb03011d0).reader couldn't read tag, (0) Success
   -70> 2014-12-19 10:04:46.514712 b07ffb40  2 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 pipe(0xb03015f8 sd=15 :6803 s=4 pgs=2 cs=1 l=1 c=0xb03011d0).fault (0) Success
   -69> 2014-12-19 10:04:46.516718 b7072740  1 -- 127.0.0.1:6803/14957 shutdown complete.
   -68> 2014-12-19 10:04:46.516838 b7072740  1 -- 127.0.0.1:0/14957 shutdown complete.
   -67> 2014-12-19 10:04:46.517050 b7072740  1 -- 127.0.0.1:0/0 learned_addr learned my addr 127.0.0.1:0/0
   -66> 2014-12-19 10:04:46.517189 b7072740  1  Processor -- bind bind my_inst.addr is 127.0.0.1:6803/14957
   -65> 2014-12-19 10:04:46.517205 b7072740  1  Processor -- start start
   -64> 2014-12-19 10:04:46.517233 b7072740  1 -- 127.0.0.1:6803/14957 start start
   -63> 2014-12-19 10:04:46.517252 b7072740  1  Processor -- start start
   -62> 2014-12-19 10:04:46.517254 b7072740  1 -- :/0 start start
   -61> 2014-12-19 10:04:46.517392 b7072740  1 Event wakeup
   -60> 2014-12-19 10:04:46.517536 b7072740  1 Event wakeup
   -59> 2014-12-19 10:04:46.517572 b12fab40  1 Event wakeup
   -58> 2014-12-19 10:04:46.517630 b46fdb40  1 -- 127.0.0.1:6803/14957 >> :/0 conn(0xb03015f8 sd=15 :6803 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=15 127.0.0.1:52577/0
   -57> 2014-12-19 10:04:46.517785 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
   -56> 2014-12-19 10:04:46.517943 b46fdb40  1 Event wakeup
   -55> 2014-12-19 10:04:46.518021 b46fdb40  2 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=15 :6803 s=STATE_ACCEPTING_READY pgs=1 cs=1 l=0).handle_connect_msg accept write reply msg done
   -54> 2014-12-19 10:04:46.518048 b4efeb40  1 Event wakeup
   -53> 2014-12-19 10:04:46.518138 b4efeb40  1 Event wakeup
   -52> 2014-12-19 10:04:46.518554 b7072740  1 Event wakeup
   -51> 2014-12-19 10:04:46.518628 b4efeb40  1 Event wakeup
   -50> 2014-12-19 10:04:46.518658 b46fdb40  1 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=15 :6803 s=STATE_OPEN pgs=1 cs=1 l=0).read_bulk Peer close file descriptor 15
   -49> 2014-12-19 10:04:46.518674 b46fdb40  1 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=15 :6803 s=STATE_OPEN pgs=1 cs=1 l=0).process read tag failed, state is STATE_OPEN
   -48> 2014-12-19 10:04:46.518680 b7072740  1 Event wakeup
   -47> 2014-12-19 10:04:46.518703 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=15 :6803 s=STATE_OPEN pgs=1 cs=1 l=0).fault server, going to standby
   -46> 2014-12-19 10:04:46.518739 b7072740  1 Event wakeup
   -45> 2014-12-19 10:04:46.518794 b12fab40  1 Event wakeup
   -44> 2014-12-19 10:04:46.518818 b46fdb40  1 -- 127.0.0.1:6803/14957 >> :/0 conn(0xb0304b10 sd=16 :6803 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=16 127.0.0.1:52578/0
   -43> 2014-12-19 10:04:46.518903 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
   -42> 2014-12-19 10:04:46.519031 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb0304b10 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept connect_seq 0 vs existing 1 state 30
   -41> 2014-12-19 10:04:46.519082 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb0304b10 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept peer reset, then tried to connect to us, replacing
   -40> 2014-12-19 10:04:46.519102 b46fdb40  1 Event wakeup
   -39> 2014-12-19 10:04:46.519144 b46fdb40  1 Event wakeup
   -38> 2014-12-19 10:04:46.519152 b46fdb40  1 Event wakeup
   -37> 2014-12-19 10:04:46.519191 b46fdb40  1 Event wakeup
   -36> 2014-12-19 10:04:46.519352 b46fdb40  1 Event wakeup
   -35> 2014-12-19 10:04:46.519379 b46fdb40  2 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=16 :6803 s=STATE_ACCEPTING_READY pgs=2 cs=2 l=0).handle_connect_msg accept write reply msg done
   -34> 2014-12-19 10:04:46.519431 b4efeb40  1 Event wakeup
   -33> 2014-12-19 10:04:46.519518 b4efeb40  1 Event wakeup
   -32> 2014-12-19 10:04:46.519695 b4efeb40  1 Event wakeup
   -31> 2014-12-19 10:04:46.519823 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=16 :6803 s=STATE_OPEN pgs=2 cs=2 l=0).process bad tag 0
   -30> 2014-12-19 10:04:46.519864 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03015f8 sd=16 :6803 s=STATE_OPEN pgs=2 cs=2 l=0).fault with nothing to send, going to standby
   -29> 2014-12-19 10:04:46.519883 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=2 cs=2 l=0).read_bulk Peer close file descriptor 14
   -28> 2014-12-19 10:04:46.519889 b4efeb40  1 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=2 cs=2 l=0).process read tag failed, state is STATE_OPEN
   -27> 2014-12-19 10:04:46.519909 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=2 cs=2 l=0).fault initiating reconnect
   -26> 2014-12-19 10:04:46.519973 b7072740  1 Event wakeup
   -25> 2014-12-19 10:04:46.519997 b46fdb40  1 Event wakeup
   -24> 2014-12-19 10:04:46.520052 b12fab40  1 Event wakeup
   -23> 2014-12-19 10:04:46.520075 b4efeb40  1 -- 127.0.0.1:6803/14957 >> :/0 conn(0xb03081e0 sd=15 :6803 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=15 127.0.0.1:52579/0
   -22> 2014-12-19 10:04:46.520124 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
   -21> 2014-12-19 10:04:46.520275 b4efeb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept we reset (peer sent cseq 3), sending RESETSESSION
   -20> 2014-12-19 10:04:46.520329 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=2 cs=3 l=0).handle_connect_replyconnect got RESETSESSION
   -19> 2014-12-19 10:04:46.520343 b4efeb40  1 Event wakeup
   -18> 2014-12-19 10:04:46.520510 b4efeb40  1 Event wakeup
   -17> 2014-12-19 10:04:46.520553 b4efeb40  2 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_ACCEPTING_READY pgs=3 cs=1 l=0).handle_connect_msg accept write reply msg done
   -16> 2014-12-19 10:04:46.520596 b4efeb40  1 Event wakeup
   -15> 2014-12-19 10:04:46.520751 b4efeb40  1 Event wakeup
   -14> 2014-12-19 10:04:46.520978 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=3 cs=1 l=0).process bad tag 0
   -13> 2014-12-19 10:04:46.521015 b4efeb40  0 -- 127.0.0.1:0/14957 >> 127.0.0.1:6803/14957 conn(0xa6c9040 sd=14 :0 s=STATE_OPEN pgs=3 cs=1 l=0).fault initiating reconnect
   -12> 2014-12-19 10:04:46.521075 b4efeb40  1 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_OPEN pgs=3 cs=1 l=0).read_bulk Peer close file descriptor 15
   -11> 2014-12-19 10:04:46.521085 b4efeb40  1 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_OPEN pgs=3 cs=1 l=0).process read tag failed, state is STATE_OPEN
   -10> 2014-12-19 10:04:46.521113 b4efeb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb03081e0 sd=15 :6803 s=STATE_OPEN pgs=3 cs=1 l=0).fault server, going to standby
    -9> 2014-12-19 10:04:46.521190 b12fab40  1 Event wakeup
    -8> 2014-12-19 10:04:46.521234 b46fdb40  1 -- 127.0.0.1:6803/14957 >> :/0 conn(0xb030b970 sd=16 :6803 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=16 127.0.0.1:52580/0
    -7> 2014-12-19 10:04:46.521323 b4efeb40  1 -- 127.0.0.1:0/14957 learned_addr learned my addr 127.0.0.1:0/14957
    -6> 2014-12-19 10:04:46.521514 b46fdb40  0 -- 127.0.0.1:6803/14957 >> 127.0.0.1:0/14957 conn(0xb030b970 sd=16 :6803 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msgaccept connect_seq 2 vs existing 1 state 30
    -5> 2014-12-19 10:04:46.521572 b46fdb40  1 Event wakeup
    -4> 2014-12-19 10:04:46.521599 b4efeb40  1 Event wakeup
    -3> 2014-12-19 10:04:46.521618 b46fdb40  1 Event wakeup
    -2> 2014-12-19 10:04:46.521645 b46fdb40  1 Event wakeup
    -1> 2014-12-19 10:04:46.521813 b7072740  1 Event wakeup
     0> 2014-12-19 10:04:46.529581 b4efeb40 -1 msg/async/AsyncConnection.cc: In function 'int AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)' thread b4efeb40 time 2014-12-19 10:04:46.521669
msg/async/AsyncConnection.cc: 1386: FAILED assert(reply.connect_seq > connect_seq)

 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xa4) [0x8a9cf05]
 2: (AsyncConnection::handle_connect_reply(ceph_msg_connect&, ceph_msg_connect_reply&)+0x854) [0x8c7b384]
 3: (AsyncConnection::_process_connection()+0x2bd9) [0x8c778b1]
 4: (AsyncConnection::process()+0x4722) [0x8c7437c]
 5: (C_handle_read::do_request(int)+0x28) [0x8c84f0c]
 6: (EventCenter::process_events(int)+0x551) [0x8c1d497]
 7: (Worker::entry()+0x212) [0x8be0c22]
 8: (Thread::entry_wrapper()+0x86) [0x8a99932]
 9: (Thread::_entry_func(void*)+0x1d) [0x8a9989f]
 10: (()+0x6d4c) [0xb7784d4c]
 11: (clone()+0x5e) [0xb7163bae]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file /home/ubuntu/ceph/src//test_out/client.admin.14957.log
--- end dump of recent events ---
2014-12-19 10:04:46.535892 b7072740 -1 *** Caught signal (Segmentation fault) **
 in thread b7072740

 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::BackTrace::BackTrace(int)+0x30) [0x893d5ba]
 2: ./unittest_msgr() [0x893cbd1]
 3: [0xb77bd400]
 4: (FakeDispatcher::Session::get_count()+0x6) [0x8915eac]
 5: (MessengerTest_StatefulTest_Test::TestBody()+0x16d9) [0x8910779]
 6: (testing::Test::Run()+0x7c) [0x8920b44]
 7: (testing::internal::TestInfoImpl::Run()+0xbb) [0x8921003]
 8: (testing::TestCase::Run()+0xb8) [0x892145c]
 9: (testing::internal::UnitTestImpl::RunAllTests()+0x258) [0x8925202]
 10: (testing::UnitTest::Run()+0x14) [0x8924390]
 11: (main()+0x120) [0x8913c71]
 12: (__libc_start_main()+0xf3) [0xb708d4d3]
 13: ./unittest_msgr() [0x890c221]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
     0> 2014-12-19 10:04:46.535892 b7072740 -1 *** Caught signal (Segmentation fault) **
 in thread b7072740

 ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
 1: (ceph::BackTrace::BackTrace(int)+0x30) [0x893d5ba]
 2: ./unittest_msgr() [0x893cbd1]
 3: [0xb77bd400]
 4: (FakeDispatcher::Session::get_count()+0x6) [0x8915eac]
 5: (MessengerTest_StatefulTest_Test::TestBody()+0x16d9) [0x8910779]
 6: (testing::Test::Run()+0x7c) [0x8920b44]
 7: (testing::internal::TestInfoImpl::Run()+0xbb) [0x8921003]
 8: (testing::TestCase::Run()+0xb8) [0x892145c]
 9: (testing::internal::UnitTestImpl::RunAllTests()+0x258) [0x8925202]
 10: (testing::UnitTest::Run()+0x14) [0x8924390]
 11: (main()+0x120) [0x8913c71]
 12: (__libc_start_main()+0xf3) [0xb708d4d3]
 13: ./unittest_msgr() [0x890c221]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

*** Caught signal (Aborted) **
 in thread b4efeb40
--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file /home/ubuntu/ceph/src//test_out/client.admin.14957.log
--- end dump of recent events ---
Actions
Copy link
 #2

Updated by Loïc Dachary over 9 years ago

  • Assignee changed from Loïc Dachary to Haomai Wang
  • Priority changed from Normal to Urgent
Actions
Copy link
 #3

Updated by Loïc Dachary over 9 years ago

Can also be reproduced anywhere with valgrind

valgrind --tool=memcheck ./unittest_msgr

Actions
Copy link
 #4

Updated by Haomai Wang over 9 years ago

  • Status changed from 12 to Fix Under Review
Actions
Copy link
 #5

Updated by Loïc Dachary over 9 years ago

  • Status changed from Fix Under Review to Resolved
  • % Done changed from 0 to 100
Actions
Copy link

Also available in: Atom PDF