Ceph

I thought I submitted this already, but I can't find the ticket for it. Sorry if its a duplicate.

With my current setup, I see this assert failure on some of my osds when I try to start everything up. I've tried with current stable, next, and master branches. This is the end of the log, followed by the stack trace.

2011-08-26 12:59:48.837602 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=41 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!
2011-08-26 12:59:48.837780 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=43 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:48.837860 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=44 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:48.994927 7f10466dc700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6809/3109 pipe(0x3d82500 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6809/10025 not 192.168.101.113:6809/3109 - wrong node!
2011-08-26 12:59:48.994958 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:48.995000 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:48.995029 7f10467dd700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6805/3036 pipe(0x3d82780 sd=20 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6805/9895 not 192.168.101.113:6805/3036 - wrong node!
2011-08-26 12:59:48.995777 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=36 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!
2011-08-26 12:59:48.995815 7f1045ed4700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6806/3511 pipe(0x3da1000 sd=41 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6806/8238 not 192.168.101.11:6806/3511 - wrong node!
2011-08-26 12:59:48.995838 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=42 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:49.038472 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.395695 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:49.395749 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:49.439074 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.857327 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:50.367145 7f104ca00700 journal throttle: waited for ops
2011-08-26 12:59:50.398605 7f104ca00700 journal throttle: waited for ops
../../src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)', in thread '0x7f104ca00700'
../../src/osd/PG.cc: 3892: FAILED assert(0 == "we got a bad state machine event")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]

• Caught signal (Aborted) *
  in thread 0x7f104ca00700
  ceph version (commit:)
  1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f4e99]
  2: /usr/ceph/bin/cosd() [0x91e3a3]
  3: (()+0xfc60) [0x7f1058cdcc60]
  4: (gsignal()+0x35) [0x7f1057462d05]
  5: (abort()+0x186) [0x7f1057466ab6]
  6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f1057d196dd]
  7: (()+0xb9926) [0x7f1057d17926]
  8: (()+0xb9953) [0x7f1057d17953]
  9: (()+0xb9a5e) [0x7f1057d17a5e]
  10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x91de9b]
  11: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
  12: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
  13: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
  14: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
  15: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
  16: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
  17: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
  18: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
  19: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
  20: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
  21: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
  22: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
  23: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
  24: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
  25: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
  26: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
  27: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
  28: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
  29: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
  30: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
  31: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
  32: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
  33: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
  34: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
  35: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
  36: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
  37: (OSD::_dispatch(Message*)+0x737) [0x832013]
  38: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
  39: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
  40: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
  41: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
  42: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
  43: (()+0x6d8c) [0x7f1058cd3d8c]
  44: (clone()+0x6d) [0x7f105751504d]

(gdb) bt
#0 0x00007f788f33ab3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000091e1f6 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x000000000091e414 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007f788dac0d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f788dac4ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f788e3776dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007f788e375926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f788e375953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f788e375a5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000091de9b in ceph::_ceph_assert_fail (assertion=0xa51798 "0 == \"we got a bad state machine event\"", file=0
xa4f709 "../../src/osd/PG.cc", line=3892, func=0xa66aa0 "PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::
RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)") at ../../src/common/assert.cc:70
#11 0x0000000000940d4c in PG::RecoveryState::Crashed::Crashed (this=0x3aa0550, ctx=...) at ../../src/osd/PG.cc:3892
#12 0x000000000097a1b3 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::shallow_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:89
#13 0x0000000000979be5 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::deep_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:79
#14 0x00000000009796ad in boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoverySta
te::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::I
nitial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct (pContext=@0x7f788305d278, outerm
ostContextBase=...) at /usr/include/boost/statechart/detail/constructor.hpp:93
#15 0x0000000000979116 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState:
:RecoveryMachine, boost::statechart::detail::no_transition_function> (this=0x3aa0550, transitionAction=...) at /usr/includ
e/boost/statechart/simple_state.hpp:804
#16 0x0000000000978a80 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed> (this=0x3aa0550) at /us
r/include/boost/statechart/simple_state.hpp:313
#17 0x000000000097863e in reactions<PG::RecoveryState::Started>::react_without_action (stt=...) at /usr/include/boost/stat
echart/transition.hpp:38
#18 0x0000000000977fe9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::without_action::react (stt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:47
#19 0x0000000000977ae9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::base::react (stt=..., evt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:68
#20 0x00000000009774aa in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::react (stt=..., evt=..., eventType=@0x7f788305d418) at /usr/include/boost/statechart/detail/reaction_dispatcher.
hpp:109
#21 0x0000000000976e70 in boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::
statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::
event_base>::no_function)>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*> (stt=..., evt=..
., eventType=@0x7f788305d418) at /usr/include/boost/statechart/transition.hpp:59
#22 0x000000000097651a in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed> >, boost::statechart::simple_
state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::histor
y_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:822
#23 0x0000000000975863 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition
<boost::statechart::event_base, PG::RecoveryState::Crashed> > > (this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/incl
ude/boost/statechart/simple_state.hpp:857
#24 0x00000000009740b8 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_b
ase, PG::RecoveryState::Crashed> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::Recover
yMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/i
nclude/boost/statechart/simple_state.hpp:826
#25 0x000000000097237f in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reac
tion<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed>

(this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:857

#26 0x000000000096f5d6 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl (this=0x3aa0550, evt=..., eventType=0xdb2370) at
#27 0x000000000096f03b in boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::Reco
veryState::Peering, (boost::statechart::history_mode)0>::react_impl (this=0x5501370, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#28 0x000000000096ec73 in boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::Reco
veryState::GetInfo, (boost::statechart::history_mode)0>::react_impl (this=0x5501320, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#29 0x000000000096e77f in boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::m
pl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl
::na, mpl::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0

::react_impl (this=0x45b6100, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:498

#30 0x00000000007ff99a in boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<vo
id>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator() (this=0x7f788305d6a0
) at /usr/include/boost/statechart/state_machine.hpp:87
#31 0x00000000007f835c in boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_functio
n<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::
event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial>
::exception_event_handler> (this=0x3aae708, action=...) at /usr/include/boost/statechart/null_exception_translator.hpp:33
#32 0x00000000007e1f53 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::send_event (this=0x3aae690, evt=...) at /usr/include
/boost/statechart/state_machine.hpp:888
#33 0x0000000000954569 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::process_event (this=0x3aae690, evt=...) at /usr/incl
ude/boost/statechart/state_machine.hpp:275
#34 0x000000000094a3e8 in PG::RecoveryState::handle_log (this=0x3aae690, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ..
/../src/osd/PG.cc:4735
#35 0x0000000000852794 in PG::handle_log (this=0x3aae000, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ../../src/osd/PG.
h:1598
#36 0x00000000008401b9 in OSD::handle_pg_log (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:4094
#37 0x0000000000832013 in OSD::_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2612
#38 0x0000000000830d34 in OSD::ms_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2427
#39 0x00000000008db45c in Messenger::ms_deliver_dispatch (this=0x2e6ca00, m=0x5a0e800) at ../../src/msg/Messenger.h:102
#40 0x00000000008c53cc in SimpleMessenger::dispatch_entry (this=0x2e6ca00) at ../../src/msg/SimpleMessenger.cc:356
#41 0x0000000000777dac in SimpleMessenger::DispatchThread::entry (this=0x2e6ce88) at ../../src/msg/SimpleMessenger.h:546
#42 0x00000000008a2c85 in Thread::_entry_func (arg=0x2e6ce88) at ../../src/common/Thread.cc:45
#43 0x00007f788f331d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#44 0x00007f788db7304d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#45 0x0000000000000000 in ?? ()