Project

General

Profile

Bug #1449

osd: FAILED assert(0 == "we got a bad state machine event")

Added by Sam Lang almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
OSD
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

I thought I submitted this already, but I can't find the ticket for it. Sorry if its a duplicate.

With my current setup, I see this assert failure on some of my osds when I try to start everything up. I've tried with current stable, next, and master branches. This is the end of the log, followed by the stack trace.

2011-08-26 12:59:48.837602 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=41 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!
2011-08-26 12:59:48.837780 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=43 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:48.837860 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=44 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:48.994927 7f10466dc700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6809/3109 pipe(0x3d82500 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6809/10025 not 192.168.101.113:6809/3109 - wrong node!
2011-08-26 12:59:48.994958 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:48.995000 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:48.995029 7f10467dd700 -- 192.168.101.13:6802/9549 >> 192.168.101.113:6805/3036 pipe(0x3d82780 sd=20 pgs=0 cs=0 l=0).connect claims to be 192.168.101.113:6805/9895 not 192.168.101.113:6805/3036 - wrong node!
2011-08-26 12:59:48.995777 7f1045fd5700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6802/3442 pipe(0x3da1280 sd=36 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6802/7971 not 192.168.101.11:6802/3442 - wrong node!
2011-08-26 12:59:48.995815 7f1045ed4700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6806/3511 pipe(0x3da1000 sd=41 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6806/8238 not 192.168.101.11:6806/3511 - wrong node!
2011-08-26 12:59:48.995838 7f1045cd2700 -- 192.168.101.13:0/9550 >> 192.168.101.11:6810/3716 pipe(0x3d82c80 sd=42 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6810/8456 not 192.168.101.11:6810/3716 - wrong node!
2011-08-26 12:59:49.038472 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.395695 7f1046ce2700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6801/4182 pipe(0x3d71000 sd=26 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/8671 not 192.168.101.111:6801/4182 - wrong node!
2011-08-26 12:59:49.395749 7f1046be1700 -- 192.168.101.13:6802/9549 >> 192.168.101.111:6805/4254 pipe(0x3d71c80 sd=25 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/8736 not 192.168.101.111:6805/4254 - wrong node!
2011-08-26 12:59:49.439074 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:49.857327 7f1045ad0700 -- 192.168.101.13:0/9550 >> 192.168.101.111:6802/4182 pipe(0x3da6c80 sd=38 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6802/8671 not 192.168.101.111:6802/4182 - wrong node!
2011-08-26 12:59:50.367145 7f104ca00700 journal throttle: waited for ops
2011-08-26 12:59:50.398605 7f104ca00700 journal throttle: waited for ops
../../src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)', in thread '0x7f104ca00700'
../../src/osd/PG.cc: 3892: FAILED assert(0 == "we got a bad state machine event")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91dd31]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
28: (OSD::_dispatch(Message*)+0x737) [0x832013]
29: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
33: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
34: (()+0x6d8c) [0x7f1058cd3d8c]
35: (clone()+0x6d) [0x7f105751504d]
  • Caught signal (Aborted) *
    in thread 0x7f104ca00700
    ceph version (commit:)
    1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f4e99]
    2: /usr/ceph/bin/cosd() [0x91e3a3]
    3: (()+0xfc60) [0x7f1058cdcc60]
    4: (gsignal()+0x35) [0x7f1057462d05]
    5: (abort()+0x186) [0x7f1057466ab6]
    6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f1057d196dd]
    7: (()+0xb9926) [0x7f1057d17926]
    8: (()+0xb9953) [0x7f1057d17953]
    9: (()+0xb9a5e) [0x7f1057d17a5e]
    10: (ceph::__ceph_assert_fail(char const
    , char const*, int, char const*)+0x1f3) [0x91de9b]
    11: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x940d4c]
    12: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x97a1b3]
    13: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x979be5]
    14: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9796ad]
    15: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x979116]
    16: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x978a80]
    17: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97863e]
    18: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x977fe9]
    19: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x977ae9]
    20: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x9774aa]
    21: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x976e70]
    22: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x97651a]
    23: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x975863]
    24: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x9740b8]
    25: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97237f]
    26: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96f5d6]
    27: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96f03b]
    28: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ec73]
    29: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96e77f]
    30: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7ff99a]
    31: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f835c]
    32: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7e1f53]
    33: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x954569]
    34: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x94a3e8]
    35: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x852794]
    36: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x8401b9]
    37: (OSD::_dispatch(Message*)+0x737) [0x832013]
    38: (OSD::ms_dispatch(Message*)+0x146) [0x830d34]
    39: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8db45c]
    40: (SimpleMessenger::dispatch_entry()+0x810) [0x8c53cc]
    41: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x777dac]
    42: (Thread::_entry_func(void*)+0x23) [0x8a2c85]
    43: (()+0x6d8c) [0x7f1058cd3d8c]
    44: (clone()+0x6d) [0x7f105751504d]

(gdb) bt
#0 0x00007f788f33ab3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000091e1f6 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x000000000091e414 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007f788dac0d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f788dac4ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f788e3776dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007f788e375926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f788e375953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f788e375a5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000091de9b in ceph::
_ceph_assert_fail (assertion=0xa51798 "0 == \"we got a bad state machine event\"", file=0
xa4f709 "../../src/osd/PG.cc", line=3892, func=0xa66aa0 "PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::
RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)") at ../../src/common/assert.cc:70
#11 0x0000000000940d4c in PG::RecoveryState::Crashed::Crashed (this=0x3aa0550, ctx=...) at ../../src/osd/PG.cc:3892
#12 0x000000000097a1b3 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::shallow_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:89
#13 0x0000000000979be5 in boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::
mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mp
l_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)
0>::deep_construct (pContext=@0x7f788305d278, outermostContextBase=...) at /usr/include/boost/statechart/state.hpp:79
#14 0x00000000009796ad in boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoverySta
te::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::I
nitial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct (pContext=@0x7f788305d278, outerm
ostContextBase=...) at /usr/include/boost/statechart/detail/constructor.hpp:93
#15 0x0000000000979116 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState:
:RecoveryMachine, boost::statechart::detail::no_transition_function> (this=0x3aa0550, transitionAction=...) at /usr/includ
e/boost/statechart/simple_state.hpp:804
#16 0x0000000000978a80 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed> (this=0x3aa0550) at /us
r/include/boost/statechart/simple_state.hpp:313
#17 0x000000000097863e in reactions<PG::RecoveryState::Started>::react_without_action (stt=...) at /usr/include/boost/stat
echart/transition.hpp:38
#18 0x0000000000977fe9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::without_action::react (stt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:47
#19 0x0000000000977ae9 in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::base::react (stt=..., evt=...) at /usr/include/boost/statechart/detail/reaction_dispatcher.hpp:68
#20 0x00000000009774aa in boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::
event_base, PG::RecoveryState::Crashed>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statech
art::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void
const*>::react (stt=..., evt=..., eventType=@0x7f788305d418) at /usr/include/boost/statechart/detail/reaction_dispatcher.
hpp:109
#21 0x0000000000976e70 in boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::
statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::
event_base>::no_function)>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*> (stt=..., evt=..
., eventType=@0x7f788305d418) at /usr/include/boost/statechart/transition.hpp:59
#22 0x000000000097651a in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed> >, boost::statechart::simple_
state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::histor
y_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:822
#23 0x0000000000975863 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition
<boost::statechart::event_base, PG::RecoveryState::Crashed> > > (this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/incl
ude/boost/statechart/simple_state.hpp:857
#24 0x00000000009740b8 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::li
st<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_b
ase, PG::RecoveryState::Crashed> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::Recover
yMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> > (stt=..., evt=..., eventType=0xdb2370) at /usr/i
nclude/boost/statechart/simple_state.hpp:826
#25 0x000000000097237f in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reac
tion<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed>

(this=0x3aa0550, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:857

#26 0x000000000096f5d6 in boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl (this=0x3aa0550, evt=..., eventType=0xdb2370) at
#27 0x000000000096f03b in boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::Reco
veryState::Peering, (boost::statechart::history_mode)0>::react_impl (this=0x5501370, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#28 0x000000000096ec73 in boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::Reco
veryState::GetInfo, (boost::statechart::history_mode)0>::react_impl (this=0x5501320, evt=..., eventType=0xdb2370) at /usr/
include/boost/statechart/simple_state.hpp:498
#29 0x000000000096e77f in boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::m
pl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl
::na, mpl::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0

::react_impl (this=0x45b6100, evt=..., eventType=0xdb2370) at /usr/include/boost/statechart/simple_state.hpp:498

#30 0x00000000007ff99a in boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<vo
id>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator() (this=0x7f788305d6a0
) at /usr/include/boost/statechart/state_machine.hpp:87
#31 0x00000000007f835c in boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_functio
n<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::
event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial>
::exception_event_handler> (this=0x3aae708, action=...) at /usr/include/boost/statechart/null_exception_translator.hpp:33
#32 0x00000000007e1f53 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::send_event (this=0x3aae690, evt=...) at /usr/include
/boost/statechart/state_machine.hpp:888
#33 0x0000000000954569 in boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial,
std::allocator<void>, boost::statechart::null_exception_translator>::process_event (this=0x3aae690, evt=...) at /usr/incl
ude/boost/statechart/state_machine.hpp:275
#34 0x000000000094a3e8 in PG::RecoveryState::handle_log (this=0x3aae690, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ..
/../src/osd/PG.cc:4735
#35 0x0000000000852794 in PG::handle_log (this=0x3aae000, from=1, msg=0x5a0e800, rctx=0x7f788305d870) at ../../src/osd/PG.
h:1598
#36 0x00000000008401b9 in OSD::handle_pg_log (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:4094
#37 0x0000000000832013 in OSD::_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2612
#38 0x0000000000830d34 in OSD::ms_dispatch (this=0x2e75000, m=0x5a0e800) at ../../src/osd/OSD.cc:2427
#39 0x00000000008db45c in Messenger::ms_deliver_dispatch (this=0x2e6ca00, m=0x5a0e800) at ../../src/msg/Messenger.h:102
#40 0x00000000008c53cc in SimpleMessenger::dispatch_entry (this=0x2e6ca00) at ../../src/msg/SimpleMessenger.cc:356
#41 0x0000000000777dac in SimpleMessenger::DispatchThread::entry (this=0x2e6ce88) at ../../src/msg/SimpleMessenger.h:546
#42 0x00000000008a2c85 in Thread::_entry_func (arg=0x2e6ce88) at ../../src/common/Thread.cc:45
#43 0x00007f788f331d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#44 0x00007f788db7304d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#45 0x0000000000000000 in ?? ()


Related issues

Duplicates Ceph - Bug #1403: osd: FAILED assert(0 == "we got a bad state machine event") Resolved 08/17/2011

History

#1 Updated by Sage Weil almost 9 years ago

  • Target version set to v0.36
  • translation missing: en.field_position set to 1

#2 Updated by Sage Weil almost 9 years ago

  • Priority changed from Normal to High

Just to confirm, this is with v0.34 (or later)?

#3 Updated by Sam Lang almost 9 years ago

I was using the stable branch, and it was definitely after v0.34 was released, but I'm not positive that there weren't other commits after v0.34 to stable for this particular run.

#4 Updated by Sage Weil almost 9 years ago

  • Subject changed from FAILED assert(0 == "we got a bad state machine event") to osd: FAILED assert(0 == "we got a bad state machine event")
  • translation missing: en.field_position deleted (8)
  • translation missing: en.field_position set to 6

#5 Updated by Samuel Just almost 9 years ago

Would it be possible to reproduce this with osd logging turned up to 20?

#6 Updated by Sam Lang almost 9 years ago

I hit this assertion a bunch of times again in my current setup/environment, but when I try to restart the failed osds (with debugging enabled, but it doesn't seem to matter), I don't see the assertion again. My environment looks like this:

  • 5 nodes, 6-8 drives per node, one osd process per drive (40 total osds).
  • Copied in ~500GB of data, with 3x replication, so around 1.5TB total used.
  • One node failed (I just shut it down and did not bring it back up) with 6 osd processes, no user I/O was happening at the time
  • Those nodes were detected as failed and marked down/out
  • I was getting client hangs, so I went ahead and marked osd[0-5] as lost
  • The pgs seemed to get stuck in the creating state, so I restarted all the osd processes
  • Some (5-6 out of 32) of them failed with the above assertion

Let me know if I can provide more info!

#7 Updated by Sage Weil almost 9 years ago

Just to clarify, have you seen this without marking osds as lost?

#8 Updated by Sam Lang almost 9 years ago

Hmm...I can't remember. I don't think I had marked the osds as lost, but I'm not 100% sure.

#9 Updated by Sam Lang almost 9 years ago

I take that back. #1403 indicates that I got the same assertion without markng the osds as lost, but it was the osds that had been restarted that got the assertion.

#10 Updated by Sage Weil almost 9 years ago

  • Target version changed from v0.36 to v0.37

#11 Updated by Josh Durgin almost 9 years ago

Replicated this with osd debugging enabled by thrashing with teuthology. Logs are in vit:~joshd/thrash_bad_state_machine_event/*/log/osd*.log.

#12 Updated by Josh Durgin almost 9 years ago

  • Status changed from New to 7

#13 Updated by Sage Weil almost 9 years ago

  • Status changed from 7 to Resolved

#14 Updated by Josh Durgin almost 9 years ago

  • Status changed from Resolved to In Progress
  • Target version changed from v0.37 to v0.38

This happened again, with a log being received in GetInfo. This was during radosbench and fast thrashing.
From teuthology:~/log/osd.6.log.gz:

osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, m
pl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history
_mode)0u>::my_context)', in thread '0x7fbc8a733700'
osd/PG.cc: 3948: FAILED assert(0 == "we got a bad state machine event")
 ceph version 0.36-285-ge07b956 (commit:e07b956b84962fd794160a54847aa4ccbcecfc89)
 1: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, 
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_contex
t)+0x14f) [0x7130af]
 2: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMac
hine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryS
tate::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::all
ocator<void>, boost::statechart::null_exception_translator>&)+0x36) [0x757116]
 3: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::eve
nt_base const&, void const*)+0x168) [0x757608]
 4: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_bas
e const&, void const*)+0x1d5) [0x757985]
 5: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_bas
e const&, void const*)+0x104) [0x759144]
 6: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::n
a, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const
&, void const*)+0x114) [0x75a394]
 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::state
chart::event_base const&)+0x8d) [0x74469d]
 8: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x204) [0x70d9d4]
 9: (OSD::handle_pg_log(MOSDPGLog*)+0x445) [0x597d85]
 10: (OSD::_dispatch(Message*)+0x773) [0x598dd3]
 11: (OSD::ms_dispatch(Message*)+0x17b) [0x599fcb]
 12: (SimpleMessenger::dispatch_entry()+0x9c2) [0x6199c2]
 13: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x4a431c]
 14: (Thread::_entry_func(void*)+0x12) [0x613ef2]
 15: (()+0x7971) [0x7fbc969b8971]
 16: (clone()+0x6d) [0x7fbc9524892d]

#15 Updated by Samuel Just almost 9 years ago

  • Status changed from In Progress to Resolved

b3c68a514135318e0dfda9f929f15f26340cd664

Also available in: Atom PDF