Ceph

Run: http://pulpito.ceph.com/teuthology-2015-09-29_05:00:01-smoke-master-distro-basic-multi/
Job: 1076136
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-09-29_05:00:01-smoke-master-distro-basic-multi/1076136/teuthology.log

2015-09-29T08:35:01.314 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.739786798155
2015-09-29T08:35:01.315 INFO:teuthology.orchestra.run.burnupi05:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 0.739786798155'
2015-09-29T08:35:01.780 INFO:teuthology.orchestra.run.burnupi05.stderr:reweighted osd.2 to 0.739787 (bd62)
2015-09-29T08:35:03.119 INFO:tasks.ceph.osd.0.burnupi05.stderr:osd/ReplicatedPG.cc: In function 'int ReplicatedPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7f71da332700 time 2015-09-29 08:35:03.105079
2015-09-29T08:35:03.120 INFO:tasks.ceph.osd.0.burnupi05.stderr:osd/ReplicatedPG.cc: 8849: FAILED assert(head_obc)
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: ceph version 9.0.3-1968-g8322a5c (8322a5c2a6b60dd2b39ee2942f79616d171b3073)
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f71fd24ddab]
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: 2: (ReplicatedPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x834) [0x7f71fce826e4]
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: 3: (ReplicatedPG::recover_primary(int, ThreadPool::TPHandle&)+0x11c3) [0x7f71fce85513]
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: 4: (ReplicatedPG::start_recovery_ops(int, ThreadPool::TPHandle&, int*)+0x53b) [0x7f71fcea141b]
2015-09-29T08:35:03.133 INFO:tasks.ceph.osd.0.burnupi05.stderr: 5: (OSD::do_recovery(PG*, ThreadPool::TPHandle&)+0x326) [0x7f71fcccc236]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: 6: (OSD::RecoveryWQ::_process(PG*, ThreadPool::TPHandle&)+0x17) [0x7f71fcd12db7]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0x7f71fd23f826]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: 8: (ThreadPool::WorkThread::entry()+0x10) [0x7f71fd2406f0]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: 9: (()+0x8182) [0x7f71fb604182]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: 10: (clone()+0x6d) [0x7f71f994b47d]
2015-09-29T08:35:03.134 INFO:tasks.ceph.osd.0.burnupi05.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.