Ceph

12> 2013-09-27 18:12:16.095353 7fecb4b52700 10 osd.0 pg_epoch: 1532 pg[3.41( v 1518'111 (1503'101,1518'111] lb d35bbc41/burnupi0946824-191/4d9//3 local-les=1527 n=0 ec=7 les/c 1527/1503 1525/1526/1091) [3,0]/[3,0,2] r=1 lpr=1527 pi=1502-1525/2 luod=0'0 active] _applied_recovered_object_replica
-11> 2013-09-27 18:12:16.095365 7fecb4b52700 1 - 10.214.134.36:6819/14670 --> 10.214.134.34:6810/59427 -- MOSDPGPushReply(3.41 1532 [PushReplyOp(d35bbc41/burnupi0946824-191/4d9//3),PushReplyOp(d35bbc41/burnupi0946824-191/head//3)]) v1 -- ?+0 0x6774800 con 0x25ea840
10> 2013-09-27 18:12:16.095391 7fecb5b54700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 287572.0.1
-9> 2013-09-27 18:12:16.095396 7fecb5b54700 10 filestore > header.spos 0.0.0
-8> 2013-09-27 18:12:16.095433 7fecb5b54700 10 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x46e05f0 seq 287572 r = 0, finisher 0x34f3bb0 0
-7> 2013-09-27 18:12:16.095439 7fecb5b54700 10 filestore(/var/lib/ceph/osd/ceph-0) _finish_op 0x46e05f0 seq 287572 osr(3.41 0x3c92fe0)/0x3c92fe0
-6> 2013-09-27 18:12:16.665578 7fecae425700 1 - 10.214.134.36:6820/14670 <== osd.1 10.214.134.36:0/21782 346 ==== osd_ping(ping e1532 stamp 2013-09-27 18:12:16.665455) v2 ==== 47+0+0 (1966344994 0 0) 0x36c7540 con 0x2a71000
5> 2013-09-27 18:12:16.665620 7fecae425700 1 - 10.214.134.36:6820/14670 --> 10.214.134.36:0/21782 -- osd_ping(ping_reply e1532 stamp 2013-09-27 18:12:16.665455) v2 -- ?+0 0x41f8a80 con 0x2a71000
4> 2013-09-27 18:12:16.665612 7fecaf427700 1 - 10.214.134.36:6821/14670 <== osd.1 10.214.134.36:0/21782 346 ==== osd_ping(ping e1532 stamp 2013-09-27 18:12:16.665455) v2 ==== 47+0+0 (1966344994 0 0) 0x26de8c0 con 0x2a712c0
3> 2013-09-27 18:12:16.665639 7fecae425700 20 osd.0 1532 _share_map_outgoing 0x25512c0 already has epoch 1532
-2> 2013-09-27 18:12:16.665651 7fecaf427700 1 - 10.214.134.36:6821/14670 --> 10.214.134.36:0/21782 -- osd_ping(ping_reply e1532 stamp 2013-09-27 18:12:16.665455) v2 -- ?+0 0x30551c0 con 0x2a712c0
-1> 2013-09-27 18:12:16.665672 7fecaf427700 20 osd.0 1532 _share_map_outgoing 0x25512c0 already has epoch 1532
0> 2013-09-27 18:12:16.722563 7fecab41f700 -1 osd/PGLog.cc: In function 'void PGLog::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int, pg_info_t&, std::list<hobject_t>&, bool&, bool&)' thread 7fecab41f700 time 2013-09-27 18:12:15.711364
osd/PGLog.cc: 402: FAILED assert(log.head >= olog.tail && olog.head >= log.tail)

ceph version 0.69-371-g99c2df9 (99c2df94e14a7994bc5284974868137dca15d5bb)
1: (PGLog::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int, pg_info_t&, std::list<hobject_t, std::allocator<hobject_t> >&, bool&, bool&)+0x1b74) [0x7d7594]
2: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int)+0x51) [0x764d61]
3: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x136) [0x79f816]
4: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::st
atechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x182) [0x7cad02]
5: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x7afb5b]
6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x11) [0x7afeb1]
7: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x303) [0x76ba43]
8: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x2be) [0x6288ce]
9: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x66fbe2]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x93fa96]
11: (ThreadPool::WorkThread::entry()+0x10) [0x9418a0]
12: (()+0x7e9a) [0x7fecbfa71e9a]
13: (clone()+0x6d) [0x7fecbde71ccd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.