Ceph

2012-07-19T15:18:36.319 INFO:teuthology.task.ceph.mds.a.err:osdc/Filer.cc: In function 'void Filer::_probed(Filer::Probe*, const object_t&, uint64_t, utime_t)' thread 7f62b34bb700 time 2012-07-19 15:18:36.331737
2012-07-19T15:18:36.319 INFO:teuthology.task.ceph.mds.a.err:osdc/Filer.cc: 161: FAILED assert(probe->known_size[p->oid] <= shouldbe)
2012-07-19T15:18:36.320 INFO:teuthology.task.ceph.mds.a.err: ceph version 0.48argonaut-466-g6e06444 (commit:6e064446b538530240fa84bac9c686d89b1bdaf7)
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 1: (Filer::_probed(Filer::Probe*, object_t const&, unsigned long, utime_t)+0xfd3) [0x6d7513]
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 2: (Objecter::C_Stat::finish(int)+0xc0) [0x6d7bd0]
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 3: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xde8) [0x6c3138]
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 4: (MDS::handle_core_message(Message*)+0xae8) [0x4cd5b8]
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 5: (MDS::_dispatch(Message*)+0x2f) [0x4cd77f]
2012-07-19T15:18:36.321 INFO:teuthology.task.ceph.mds.a.err: 6: (MDS::ms_dispatch(Message*)+0x20b) [0x4cf36b]
2012-07-19T15:18:36.322 INFO:teuthology.task.ceph.mds.a.err: 7: (DispatchQueue::entry()+0x6b1) [0x7d1ee1]
2012-07-19T15:18:36.322 INFO:teuthology.task.ceph.mds.a.err: 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x75d7cd]
2012-07-19T15:18:36.322 INFO:teuthology.task.ceph.mds.a.err: 9: (()+0x7e9a) [0x7f62b8014e9a]
2012-07-19T15:18:36.322 INFO:teuthology.task.ceph.mds.a.err: 10: (clone()+0x6d) [0x7f62b67cd4bd]

i bet this is another case of out-of-order osd replies.

ubuntu@teuthology:/a/sage-2012-07-19_15:03:51-regression-wip-msgr-cleanup-testing-basic/14347$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 14240f8208136dbbe7e825caedc0104806027aae
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 6e064446b538530240fa84bac9c686d89b1bdaf7
  workunit:
    sha1: 6e064446b538530240fa84bac9c686d89b1bdaf7
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana12.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDre5Wd3DbprwoJYrUe48niK0uhqzlfbntgARcodsY1iV42HW1zyFlvhxbg/V2S4X7Bo1Ymu9PVyTXoRDhQlBqqWv3BbEsk6FLf7QGdDS5wRRIHCBdu9eeB+8FmG5xf9akx/U0hR6TozlMHMRs4s/DUS3heG1oT1o+aaDo77fI4GlieifEgE8DjOA7fhVnjh3L39ZVaIgps9DGvmdlhSPOMaN3ENNDzSJBoUDGWGNMw7Jurwu06qGgM7d36IUISqbO61FZZuhHXQhFAG0s1gHOusALw8uHOOUv6+ClCNw+h4KYyCiPqK3liJZyInHgD64hDf51dA8Q73mKh2pnrwxKT
  ubuntu@plana13.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBCv+O3jMwlV+Cu2/+fryD4y6zHcLSJYHD47MVt+7oCMuYLDRHIeezoege4H10/DVy8KjEjMrft8KtFKPMHbfvVvgVYeb8qEF5w7GfFxMz1ox2ThT1heEPtrpBqdF9p2lb6aS+S2tC5noTyb0qoVci6nUK3A3cl+LTh+n+skHviMlJok3tyqz6Ye/j011i4pfiPNbuwR7WKUuQs8hhoDy0pztzbhTZE+KZ42LKM34t9hB1NFT1uVRvfiCMaq4e+SdcwuZRlqws+LG/KpJ/5wsmBhjhBMbDXzks8PDBpnmiKJ5cPPYDmJ4QtAUEdjb6B8awli/R3EGYl0D8RolY9sR1
  ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuMOcu2XPQovy/Qzmwyvc9tvGP9JZVJ6cqiJ3RPOSGgAifKLTxe2ramHpD8AKcdthu8VAfouFpZK4CtBWKJowurR+4yZKgEugzvYuZ/nK/np56vreBQmRBWD1vLPtxPsTT3YGu5qx+ixdSwrSxexxc0/7+EW9x1D6knL+OGUNWksoGIRlXxjh9qafbw/1XKeQQF28vxBXHofXUFY8USMUcq5HDuaFfmgKzufH6vk84oqyr/jtGej6b4g6tbGiHPYR+o5tmTQHyxpOxqLZP2RFFqHlQ/QaOmRvSNIoOo+1UbqdcWsLk16/lXIS1mI+BZsZouk1H+fGeMTEUDGktiPW7
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/fsstress.sh