Ceph

OS: Ubuntu 12.04
Release: 0.56.3-1precise from ceph.com
Filesystem: XFS

On a brand new ceph cluster with 2 osd on a single server, a RDB-backed kvm virtual machine was created and then 2 new servers were added to the cluster, each with 1 monitor and 2 osd. That's when osd started to crash on all three servers during data rebalancing.

Complete logfiles are available on request if necessary.

osd/ReplicatedPG.cc: 5378: FAILED assert(pi.recovery_info.soid.snap != ((__u64)(-2)) && pi.recovery_info.soid.snap != ((__u64)(-1)))

 ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
 1: /usr/bin/ceph-osd() [0x5b55d1]
 2: (ReplicatedPG::sub_op_push(std::tr1::shared_ptr<OpRequest>)+0xad) [0x5b568d]
 3: (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0x537) [0x5b5c77]
 4: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x1bc) [0x6834ac]
 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>)+0x323) [0x5ec2a3]
 6: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>)+0x49b) [0x601a2b]
 7: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_process(boost::intrusive_ptr<PG>, Thread
Pool::TPHandle&)+0x31) [0x63a031]
 8: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle
&)+0x9c) [0x63a25c]
 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x823cf6]
 10: (ThreadPool::WorkThread::entry()+0x10) [0x825b20]
 11: (()+0x7e9a) [0x7ff40755ce9a]
 12: (clone()+0x6d) [0x7ff405fe0cbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.