Ceph

This occurred on both osds during a teuthology run of s3-tests.

$ zcat ~teuthworker/archive/nightly_coverage_2011-09-05/650/remote/ubuntu@sepia48.ceph.dreamhost.com/log/osd.0.log.gz | tail -n 46
os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&)', in thread '0x7f15ebdd6700'
os/FileStore.cc: 2390: FAILED assert(0 == "ENOENT on clone suggests osd bug")
 ceph version 0.34-409-g479e9ca (commit:479e9ca674e4271a1635c59ef3680e9ab8b89f73)
 1: (FileStore::_do_transaction(ObjectStore::Transaction&)+0xae6) [0x7b0876]
 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0xa5) [0x7b60b5]
 3: (FileStore::_do_op(FileStore::OpSequencer*)+0x26b) [0x7a681b]
 4: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x15) [0x7c3af5]
 5: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x12) [0x7b7df2]
 6: (ThreadPool::worker()+0x846) [0x661976]
 7: (ThreadPool::WorkThread::entry()+0x15) [0x5a5245]
 8: (Thread::_entry_func(void*)+0x12) [0x608d82]
 9: (()+0x7971) [0x7f15f5055971]
 10: (clone()+0x6d) [0x7f15f38e592d]
 ceph version 0.34-409-g479e9ca (commit:479e9ca674e4271a1635c59ef3680e9ab8b89f73)
 1: (FileStore::_do_transaction(ObjectStore::Transaction&)+0xae6) [0x7b0876]
 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0xa5) [0x7b60b5]
 3: (FileStore::_do_op(FileStore::OpSequencer*)+0x26b) [0x7a681b]
 4: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x15) [0x7c3af5]
 5: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x12) [0x7b7df2]
 6: (ThreadPool::worker()+0x846) [0x661976]
 7: (ThreadPool::WorkThread::entry()+0x15) [0x5a5245]
 8: (Thread::_entry_func(void*)+0x12) [0x608d82]
 9: (()+0x7971) [0x7f15f5055971]
 10: (clone()+0x6d) [0x7f15f38e592d]
*** Caught signal (Aborted) **
 in thread 0x7f15ebdd6700
 ceph version 0.34-409-g479e9ca (commit:479e9ca674e4271a1635c59ef3680e9ab8b89f73)
 1: /tmp/cephtest/binary/usr/local/bin/cosd() [0x6766b4]
 2: (()+0xfb40) [0x7f15f505db40]
 3: (gsignal()+0x35) [0x7f15f3832ba5]
 4: (abort()+0x180) [0x7f15f38366b0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f15f40d66bd]
 6: (()+0xb9906) [0x7f15f40d4906]
 7: (()+0xb9933) [0x7f15f40d4933]
 8: (()+0xb9a3e) [0x7f15f40d4a3e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x52f) [0x60a1bf]
 10: (FileStore::_do_transaction(ObjectStore::Transaction&)+0xae6) [0x7b0876]
 11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0xa5) [0x7b60b5]
 12: (FileStore::_do_op(FileStore::OpSequencer*)+0x26b) [0x7a681b]
 13: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x15) [0x7c3af5]
 14: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x12) [0x7b7df2]
 15: (ThreadPool::worker()+0x846) [0x661976]
 16: (ThreadPool::WorkThread::entry()+0x15) [0x5a5245]
 17: (Thread::_entry_func(void*)+0x12) [0x608d82]
 18: (()+0x7971) [0x7f15f5055971]
 19: (clone()+0x6d) [0x7f15f38e592d]