Ceph

figured out where the thrasher errors are coming from:

2013-01-30T10:48:02.890 INFO:teuthology.task.thrashosds.thrasher:in_osds:  [1, 2, 0, 4]  out_osds:  [3, 5] dead_osds:  [3] live_osds:  [1, 0, 2, 5, 4]
2013-01-30T10:48:02.890 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 3
2013-01-30T10:48:02.890 INFO:teuthology.task.ceph.osd.3:Restarting
2013-01-30T10:48:02.891 DEBUG:teuthology.orchestra.run:Running: '/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper term /tmp/cephtest/binary/usr/local/bin/ceph-osd -f -i 3 -c 
/tmp/cephtest/ceph.conf'
2013-01-30T10:48:02.894 INFO:teuthology.task.ceph.osd.3:Started
2013-01-30T10:48:02.894 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete
2013-01-30T10:48:02.894 DEBUG:teuthology.orchestra.run:Running: 'LD_LIBRARY_PRELOAD=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin
/ceph -k /tmp/cephtest/ceph.keyring -c /tmp/cephtest/ceph.conf --concise -- pg dump --format=json'
2013-01-30T10:48:02.947 DEBUG:teuthology.orchestra.run:Running: 'LD_LIBRARY_PRELOAD=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin
/ceph -k /tmp/cephtest/ceph.keyring -c /tmp/cephtest/ceph.conf --concise -- pg dump --format=json'
2013-01-30T10:48:02.948 INFO:teuthology.task.ceph.osd.3.out:starting osd.3 at :/0 osd_data /tmp/cephtest/data/osd.3.data /tmp/cephtest/data/osd.3.journal
2013-01-30T10:48:03.027 INFO:teuthology.task.ceph.osd.3.err:2013-01-30 10:47:47.704046 7f8a55f09780 -1 filestore(/tmp/cephtest/data/osd.3.data) limited size xattrs -- filestore_xattr_use_omap enabled
2013-01-30T10:48:03.037 DEBUG:teuthology.orchestra.run:Running: 'LD_LIBRARY_PRELOAD=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin
/ceph -k /tmp/cephtest/ceph.keyring -c /tmp/cephtest/ceph.conf --concise -s'
2013-01-30T10:48:03.108 INFO:teuthology.task.thrashosds.ceph_manager:   health HEALTH_OK
   monmap e1: 3 mons at {a=10.214.131.15:6789/0,b=10.214.131.15:6790/0,c=10.214.131.24:6789/0}, election epoch 6, quorum 0,1,2 a,b,c
   osdmap e104: 6 osds: 5 up, 4 in
    pgmap v142: 72 pgs: 72 active+clean; 108 MB data, 1519 MB used, 2609 GB / 2750 GB avail
   mdsmap e5: 1/1/1 up {0=a=up:active}

2013-01-30T10:48:03.108 INFO:teuthology.task.thrashosds.ceph_manager:recovered!
2013-01-30T10:48:03.108 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0
2013-01-30T10:48:03.112 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3
2013-01-30T10:48:03.112 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3
2013-01-30T10:48:03.112 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False

i'm guessing it's just that osd.3 hasn't started up yet, causing the ceph tell osd.3 to fail.