Ceph

2013-11-05 17:30:54.916068 7f9f42a66700 1 osd/PGLog.cc: In function 'void PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t, pg_info_t&, std::list<hobject_t>&, bool&, bool&)' thread 7f9f42a66700 time 2013-11-05 17:30:54.913958
osd/PGLog.cc: 368: FAILED assert(p>version > newhead)

ceph version 0.72-rc1-30-gd7202fd (d7202fd672f00a1e6170ca0797f3e985b86b9c66)
 1: (PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t, pg_info_t&, std::list<hobject_t, std::allocator<hobject_t> >&, bool&, bool&)+0x3fc) [0x7d480c]
 2: (PGLog::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int, pg_info_t&, std::list<hobject_t, std::allocator<hobject_t> >&, bool&, bool&)+0x16e1) [0x7d5f51]
 3: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int)+0x51) [0x762ec1]
 4: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x136) [0x79d1e6]
 5: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::st
atechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x182) [0x7c8312]
 6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x7ad63b]
 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x11) [0x7ad991]
 8: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x303) [0x7694e3]
 9: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x264) [0x6211d4]
 10: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x668132]
 11: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x9410a6]
 12: (ThreadPool::WorkThread::entry()+0x10) [0x942eb0]
 13: (()+0x7e9a) [0x7f9f57b06e9a]
 14: (clone()+0x6d) [0x7f9f55ebfccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

This is actually pretty simple. A few lines up, we need to account for the possibility that the log does not contain an entry at log.head.