Ceph

     0> 2013-06-16 16:33:51.226048 7f2b6d9ec700 -1 osd/SnapMapper.cc: In function 'int SnapMapper::remove_oid(const hobject_t&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7f2b6d9ec700 time 2013-06-16 16:33:51.223668
osd/SnapMapper.cc: 270: FAILED assert(check(oid))

 ceph version 0.64-468-g80c2d5a (80c2d5a8db0900e727275810ac8d160ababb82ae)
 1: ceph-osd() [0x6dd7b2]
 2: (remove_dir(ObjectStore*, SnapMapper*, OSDriver*, ObjectStore::Sequencer*, coll_t, std::tr1::shared_ptr<DeletingState>)+0x3e1) [0x6564a1]
 3: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> >)+0x2d0) [0x656f60]
 4: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> > >::_process(std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> >, ThreadPool
::TPHandle&)+0x52) [0x6a4d52]
 5: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0xd4) [0x6a4e84]
 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x8a4266]
 7: (ThreadPool::WorkThread::entry()+0x10) [0x8a6090]
 8: (()+0x7e9a) [0x7f2b81389e9a]
 9: (clone()+0x6d) [0x7f2b7f51cccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2013-06-16_15:29:52-rados-master-testing-basic/37315$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: f28d6bf9ef4d3f8dd2e669fd21be827663f215e2
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject socket failures: 2500
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
    fs: xfs
    log-whitelist:
    - slow request
    sha1: 80c2d5a8db0900e727275810ac8d160ababb82ae
  install:
    ceph:
      sha1: 80c2d5a8db0900e727275810ac8d160ababb82ae
  s3tests:
    branch: master
  workunit:
    sha1: 80c2d5a8db0900e727275810ac8d160ababb82ae
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000