Ceph

2013-02-18 20:54:39.562581 7f696abcc700 -1 osd/ReplicatedPG.cc: In function 'int ReplicatedPG::pull(const hobject_t&, eversion_t, int)' thread 7f696abcc700 time 2013-02-18 20:54:39.560108
osd/ReplicatedPG.cc: 4994: FAILED assert(log.objects.count(soid) && log.objects[soid]->op == pg_log_entry_t::LOST_REVERT && log.objects[soid]->reverting_to == v)

 ceph version 0.56-1038-g1d96bf1 (1d96bf1965dc9b1d9b2aa1b3e2315fab6bdd2e08)
 1: (ReplicatedPG::pull(hobject_t const&, eversion_t, int)+0xbc3) [0x5aa013]
 2: (ReplicatedPG::pull(hobject_t const&, eversion_t, int)+0x11cf) [0x5aa61f]
 3: (ReplicatedPG::recover_primary(int)+0x1723) [0x5abf83]
 4: (ReplicatedPG::start_recovery_ops(int, PG::RecoveryCtx*)+0xff) [0x5ca87f]
 5: (OSD::do_recovery(PG*)+0x3a7) [0x62a657]
 6: (OSD::RecoveryWQ::_process(PG*)+0xd) [0x666a9d]
 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x833bc6]
 8: (ThreadPool::WorkThread::entry()+0x10) [0x8359f0]
 9: (()+0x7e9a) [0x7f697c35be9a]
 10: (clone()+0x6d) [0x7f697a6f74bd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

ubuntu@teuthology:/a/sage-ooo4/5138$ cat orig.config.yaml 
nuke-on-error: true
overrides:
  ceph:
    branch: wip-2947
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject socket failures: 2500
      osd:
        debug ms: 1
    fs: xfs
    log-whitelist:
    - slow request
roles:
- - mon.a
  - mon.b
  - osd.0
  - osd.1
  - osd.2
- - mon.c
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000