Ceph

ubuntu@teuthology:/a/gregf-2014-05-29_16:19:17-rados-wip-client-fast-dispatch-testing-basic-plana/280831/remote/plana47/log/ceph-osd.4.log.gz

2014-05-30 16:18:52.985619 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.4s2
2014-05-30 16:18:52.985658 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.6s2
2014-05-30 16:18:52.985684 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.7s0
2014-05-30 16:18:52.985752 7fabe98f5700  1 -- 10.214.132.31:6802/7429 <== osd.5 10.214.132.31:6812/7433 2868 ==== MOSDECSubOpRead(3.4s2 262 ECSubRead(tid=1192, to_read={193d8934/plana4023310-290/head//3=526336,526336}, attrs_to_read=)) v1 ==== 136+0+0 (2632774283 0 0) 0x4265b40 con 0x34656e0
2014-05-30 16:18:52.985862 7fabe98f5700 10 osd.4 263 handle_replica_op MOSDECSubOpRead(3.4s2 262 ECSubRead(tid=1192, to_read={193d8934/plana4023310-290/head//3=526336,526336}, attrs_to_read=)) v1 epoch 262
2014-05-30 16:18:52.985875 7fabe98f5700 20 osd.4 263 should_share_map osd.5 10.214.132.31:6812/7433 262
2014-05-30 16:18:52.987841 7fabf9c18700 -1 osd/OSD.cc: In function 'void OSD::consume_map()' thread 7fabf9c18700 time 2014-05-30 16:18:52.985751
osd/OSD.cc: 6222: FAILED assert(p->second.empty())

Looks like a straightforward race that I missed with the fast dispatch changes (the OSD clears out the queue, then a fast dispatch thread delivers another op to the queue, then the OSD asserts that the queue is empty). Hopefully it will be straightforward to fix.