Bug #1449
closedosd: FAILED assert(0 == "we got a bad state machine event")
0%
Description
I thought I submitted this already, but I can't find the ticket for it. Sorry if its a duplicate.
With my current setup, I see this assert failure on some of my osds when I try to start everything up. I've tried with current stable, next, and master branches. This is the end of the log, followed by the stack trace.
2011-08-26 12:59:48.837602 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=41 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!2011-08-26 12:59:48.837780 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=43 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:48.837860 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=44 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:48.994927 7f10466dc700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6809/3109 pipe(0x3d82500 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6809/10025 not 192.168.101.113:6809/3109 - wrong node!
2011-08-26 12:59:48.994958 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:48.995000 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:48.995029 7f10467dd700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6805/3036 pipe(0x3d82780 sd=20 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6805/9895 not 192.168.101.113:6805/3036 - wrong node!
2011-08-26 12:59:48.995777 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=36 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!
2011-08-26 12:59:48.995815 7f1045ed4700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6806/3511 pipe(0x3da1000 sd=41 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6806/8238 not 192.168.101.11:6806/3511 - wrong node!
2011-08-26 12:59:48.995838 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=42 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:49.038472 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.395695 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:49.395749 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:49.439074 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.857327 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:50.367145 7f104ca00700 journal throttle: waited for ops
2011-08-26 12:59:50.398605 7f104ca00700 journal throttle: waited for ops
../../src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)', in thread '0x7f104ca00700'
../../src/osd/PG.cc: 3892: FAILED assert(0 == "we got a bad state machine event")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]
- Caught signal (Aborted) *
in thread 0x7f104ca00700
ceph version (commit:)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f4e99]
2: /usr/ceph/bin/cosd() [0x91e3a3]
3: (()+0xfc60) [0x7f1058cdcc60]
4: (gsignal()+0x35) [0x7f1057462d05]
5: (abort()+0x186) [0x7f1057466ab6]
6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f1057d196dd]
7: (()+0xb9926) [0x7f1057d17926]
8: (()+0xb9953) [0x7f1057d17953]
9: (()+0xb9a5e) [0x7f1057d17a5e]
10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x91de9b]
11: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
12: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
13: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
14: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
15: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
16: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
17: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
18: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
19: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
20: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
21: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
22: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
23: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
24: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
25: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
26: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
27: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
28: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
29: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
30: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
31: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
32: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
33: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
34: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
35: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
36: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
37: (OSD::_dispatch(Message*)+0x737) [0x832013]
38: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
39: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
40: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
41: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
42: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
43: (()+0x6d8c) [0x7f1058cd3d8c]
44: (clone()+0x6d) [0x7f105751504d]
(gdb) bt
#0 0x00007f788f33ab3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000091e1f6 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x000000000091e414 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007f788dac0d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f788dac4ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f788e3776dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007f788e375926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f788e375953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f788e375a5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000091de9b in ceph::_ceph_assert_fail (assertion=0xa51798 "0 == \"we got a bad state machine event\"", file=0
xa4f709 "../../src/osd/PG.cc", line=3892, func=0xa66aa0 "PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::
RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)") at ../../src/common/assert.cc:70
#11 0x0000000000940d4c in PG::RecoveryState::Crashed::Crashed (this=0x3aa0550, ctx=...) at ../../src/osd/PG.cc:3892
#12 0x000000000097a1b3 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::shallow_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:89
#13 0x0000000000979be5 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::deep_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:79
#14 0x00000000009796ad in boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoverySta
te::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::I
nitial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct (pContext=@0x7f788305d278, outerm
ostContextBase=...) at /usr/include/boost/statechart/detail/constructor.hpp:93
#15 0x0000000000979116 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState:
:RecoveryMachine, boost::statechart::detail::no_transition_function> (this=0x3aa0550, transitionAction=...) at /usr/includ
e/boost/statechart/simple_state.hpp:804
#16 0x0000000000978a80 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed> (this=0x3aa0550) at /us
r/include/boost/statechart/simple_state.hpp:313
#17 0x000000000097863e in reactions<PG::RecoveryState::Started>::react_without_action (stt=...) at /usr/include/boost/stat
echart/transition.hpp:38
#18 0x0000000000977fe9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::without_action::react (stt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:47
#19 0x0000000000977ae9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::base::react (stt=..., evt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:68
#20 0x00000000009774aa in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::react (stt=..., evt=..., eventType=@0x7f788305d418) at /usr/include/boost/statechart/detail/reaction_dispatcher.
hpp:109
#21 0x0000000000976e70 in boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::
statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::
event_base>::no_function)>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*> (stt=..., evt=..
., eventType=@0x7f788305d418) at /usr/include/boost/statechart/transition.hpp:59
#22 0x000000000097651a in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed> >, boost::statechart::simple_
state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::histor
y_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:822
#23 0x0000000000975863 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition
<boost::statechart::event_base, PG::RecoveryState::Crashed> > > (this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/incl
ude/boost/statechart/simple_state.hpp:857
#24 0x00000000009740b8 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_b
ase, PG::RecoveryState::Crashed> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::Recover
yMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/i
nclude/boost/statechart/simple_state.hpp:826
#25 0x000000000097237f in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reac
tion<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed>
(this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:857
#26 0x000000000096f5d6 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl (this=0x3aa0550, evt=..., eventType=0xdb2370) at
#27 0x000000000096f03b in boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::Reco
veryState::Peering, (boost::statechart::history_mode)0>::react_impl (this=0x5501370, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#28 0x000000000096ec73 in boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::Reco
veryState::GetInfo, (boost::statechart::history_mode)0>::react_impl (this=0x5501320, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#29 0x000000000096e77f in boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::m
pl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl
::na, mpl::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0
::react_impl (this=0x45b6100, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:498
#30 0x00000000007ff99a in boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<vo
id>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator() (this=0x7f788305d6a0
) at /usr/include/boost/statechart/state_machine.hpp:87
#31 0x00000000007f835c in boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_functio
n<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::
event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial>
::exception_event_handler> (this=0x3aae708, action=...) at /usr/include/boost/statechart/null_exception_translator.hpp:33
#32 0x00000000007e1f53 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::send_event (this=0x3aae690, evt=...) at /usr/include
/boost/statechart/state_machine.hpp:888
#33 0x0000000000954569 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::process_event (this=0x3aae690, evt=...) at /usr/incl
ude/boost/statechart/state_machine.hpp:275
#34 0x000000000094a3e8 in PG::RecoveryState::handle_log (this=0x3aae690, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ..
/../src/osd/PG.cc:4735
#35 0x0000000000852794 in PG::handle_log (this=0x3aae000, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ../../src/osd/PG.
h:1598
#36 0x00000000008401b9 in OSD::handle_pg_log (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:4094
#37 0x0000000000832013 in OSD::_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2612
#38 0x0000000000830d34 in OSD::ms_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2427
#39 0x00000000008db45c in Messenger::ms_deliver_dispatch (this=0x2e6ca00, m=0x5a0e800) at ../../src/msg/Messenger.h:102
#40 0x00000000008c53cc in SimpleMessenger::dispatch_entry (this=0x2e6ca00) at ../../src/msg/SimpleMessenger.cc:356
#41 0x0000000000777dac in SimpleMessenger::DispatchThread::entry (this=0x2e6ce88) at ../../src/msg/SimpleMessenger.h:546
#42 0x00000000008a2c85 in Thread::_entry_func (arg=0x2e6ce88) at ../../src/common/Thread.cc:45
#43 0x00007f788f331d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#44 0x00007f788db7304d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#45 0x0000000000000000 in ?? ()
Updated by Sage Weil over 12 years ago
- Target version set to v0.36
- Translation missing: en.field_position set to 1
Updated by Sage Weil over 12 years ago
- Priority changed from Normal to High
Just to confirm, this is with v0.34 (or later)?
Updated by Sam Lang over 12 years ago
I was using the stable branch, and it was definitely after v0.34 was released, but I'm not positive that there weren't other commits after v0.34 to stable for this particular run.
Updated by Sage Weil over 12 years ago
- Subject changed from FAILED assert(0 == "we got a bad state machine event") to osd: FAILED assert(0 == "we got a bad state machine event")
- Translation missing: en.field_position deleted (
8) - Translation missing: en.field_position set to 6
Updated by Samuel Just over 12 years ago
Would it be possible to reproduce this with osd logging turned up to 20?
Updated by Sam Lang over 12 years ago
I hit this assertion a bunch of times again in my current setup/environment, but when I try to restart the failed osds (with debugging enabled, but it doesn't seem to matter), I don't see the assertion again. My environment looks like this:
- 5 nodes, 6-8 drives per node, one osd process per drive (40 total osds).
- Copied in ~500GB of data, with 3x replication, so around 1.5TB total used.
- One node failed (I just shut it down and did not bring it back up) with 6 osd processes, no user I/O was happening at the time
- Those nodes were detected as failed and marked down/out
- I was getting client hangs, so I went ahead and marked osd[0-5] as lost
- The pgs seemed to get stuck in the creating state, so I restarted all the osd processes
- Some (5-6 out of 32) of them failed with the above assertion
Let me know if I can provide more info!
Updated by Sage Weil over 12 years ago
Just to clarify, have you seen this without marking osds as lost?
Updated by Sam Lang over 12 years ago
Hmm...I can't remember. I don't think I had marked the osds as lost, but I'm not 100% sure.
Updated by Sam Lang over 12 years ago
I take that back. #1403 indicates that I got the same assertion without markng the osds as lost, but it was the osds that had been restarted that got the assertion.
Updated by Sage Weil over 12 years ago
- Target version changed from v0.36 to v0.37
Updated by Josh Durgin over 12 years ago
Replicated this with osd debugging enabled by thrashing with teuthology. Logs are in vit:~joshd/thrash_bad_state_machine_event/*/log/osd*.log.
Updated by Josh Durgin over 12 years ago
- Status changed from New to 7
This should be fixed with b344d7979b8e0aac931c9f337f367e400a932e36
Updated by Josh Durgin over 12 years ago
- Status changed from Resolved to In Progress
- Target version changed from v0.37 to v0.38
This happened again, with a log being received in GetInfo. This was during radosbench and fast thrashing.
From teuthology:~teuthworker/archive/nightly_coverage_2011-10-12/443/remote/ubuntu@sepia27.ceph.dreamhost.com/log/osd.6.log.gz:
osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, m pl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history _mode)0u>::my_context)', in thread '0x7fbc8a733700' osd/PG.cc: 3948: FAILED assert(0 == "we got a bad state machine event") ceph version 0.36-285-ge07b956 (commit:e07b956b84962fd794160a54847aa4ccbcecfc89) 1: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_contex t)+0x14f) [0x7130af] 2: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMac hine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryS tate::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::all ocator<void>, boost::statechart::null_exception_translator>&)+0x36) [0x757116] 3: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::eve nt_base const&, void const*)+0x168) [0x757608] 4: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_bas e const&, void const*)+0x1d5) [0x757985] 5: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_bas e const&, void const*)+0x104) [0x759144] 6: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::n a, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const &, void const*)+0x114) [0x75a394] 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::state chart::event_base const&)+0x8d) [0x74469d] 8: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x204) [0x70d9d4] 9: (OSD::handle_pg_log(MOSDPGLog*)+0x445) [0x597d85] 10: (OSD::_dispatch(Message*)+0x773) [0x598dd3] 11: (OSD::ms_dispatch(Message*)+0x17b) [0x599fcb] 12: (SimpleMessenger::dispatch_entry()+0x9c2) [0x6199c2] 13: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x4a431c] 14: (Thread::_entry_func(void*)+0x12) [0x613ef2] 15: (()+0x7971) [0x7fbc969b8971] 16: (clone()+0x6d) [0x7fbc9524892d]
Updated by Samuel Just over 12 years ago
- Status changed from In Progress to Resolved
b3c68a514135318e0dfda9f929f15f26340cd664