Ceph

-1222> 2013-05-07 16:42:41.796117 7fcb871c1700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys meta/16ef7597/infos/head//-1
-1221> 2013-05-07 16:42:41.796115 7fcb7f9b2700 10 osd.3 pg_epoch: 253 pg[3.3b( v 214'17 (0'0,214'17] local-les=220 n=2 ec=6 les/c 220/232 247/247/6) [4,1] r=-1 lpr=247 pi=119-246/4 inactive NOTIFY] take_waiters
-1220> 2013-05-07 16:42:41.796128 7fcb7f9b2700 15 osd.3 pg_epoch: 253 pg[3.3b( v 214'17 (0'0,214'17] local-les=220 n=2 ec=6 les/c 220/232 247/247/6) [4,1] r=-1 lpr=247 pi=119-246/4 inactive NOTIFY] requeue_ops
-1219> 2013-05-07 16:42:41.796110 7fcb7f1b1700 -1 osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::do_op(OpRequestRef)' thread 7fcb7f1b1700 time 2013-05-07 16:42:41.775495
osd/ReplicatedPG.cc: 1078: FAILED assert(0 == "out of order op")

ceph version 0.61-126-g9e4ea8b (9e4ea8bf7ab11ad287b23d27073bec3d05c67d9c)
1: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>)+0x4256) [0x5cc2f6]
2: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x573) [0x6b0073]
3: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>)+0x323) [0x605523]
4: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>)+0x49b) [0x61d23b]
5: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x31) [0x658ed1]
6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0x9c) [0x65923c]
7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x838166]
8: (ThreadPool::WorkThread::entry()+0x10) [0x839f90]
9: (()+0x7e9a) [0x7fcb90349e9a]
10: (clone()+0x6d) [0x7fcb8e4dccbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

The sequence was:

client.4208.0:2378 (e252) arrives, object is degraded
client.4208.0:2379 (e253) arrives, waits for map
client.4208.0:2378 (e252) is requeued after recovery
client.4208.0:2379 (e253) is requeued on map arrival
client.4208.0:2379 is processed
client.4208.0:2378 is processed