Ceph » Linux kernel client

Seen duing rbd:thrah Can reprodouce 1/3 runs

Linux magna079 3.10.0-229.9.1.el7.x86_64 #1 SMP Sat Jul 4 17:21:43 EDT 2015 x86_64 x86_64 x86_64 GNU/Linu

2015-08-12T20:37:51.883 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.884 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.885 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.886 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.887 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.887 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.888 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.889 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.890 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.890 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.891 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.892 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.893 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.893 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:37:51.880548 7f34899e1700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f34899e1700 time 2015-08-12 20:37:51.878724
2015-08-12T20:37:51.894 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.895 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.896 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.896 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.897 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.898 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.898 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.899 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.900 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.901 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.901 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.902 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.903 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.904 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.905 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.957 INFO:tasks.ceph.osd.4.magna078.stderr:  -209> 2015-08-12 20:37:51.878713 7f34899e1700 -1 osd.4 pg_epoch: 197 pg[0.e( v 194'11837 (106'8625,194'11837] local-les=181 n=198 ec=1 les/c 181/165 180/180/173) [3,5]/[4,5] r=0 lpr=180 pi=164-179/4 rops=5 bft=3 crt=188'11821 lcod 194'11836 mlcod 194'11836 active+remapped+backfilling] bad op order, already applied 199013 > this 198728
2015-08-12T20:37:51.958 INFO:tasks.ceph.osd.4.magna078.stderr:   -21> 2015-08-12 20:37:51.880548 7f34899e1700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f34899e1700 time 2015-08-12 20:37:51.878724
2015-08-12T20:37:51.959 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.960 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.961 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.962 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.962 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.963 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.964 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.965 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.966 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.967 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.968 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.968 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.969 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.970 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.971 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.973 INFO:tasks.ceph.osd.4.magna078.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-08-12T20:37:51.973 INFO:tasks.ceph.osd.4.magna078.stderr:*** Caught signal (Aborted) **
2015-08-12T20:37:51.974 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:51.975 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.975 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:51.976 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:51.977 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:51.977 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:51.978 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:51.979 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:51.979 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:51.980 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:51.981 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:51.981 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.982 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.983 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.983 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.984 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.985 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.985 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.986 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.986 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.987 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:37:51.961266 7f34899e1700 -1 *** Caught signal (Aborted) **
2015-08-12T20:37:51.987 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:51.988 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.989 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.989 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:51.990 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:51.990 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:51.991 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:51.992 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:51.992 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:51.993 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:51.993 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:51.994 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:51.994 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.995 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.996 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.996 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.997 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.997 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.998 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.998 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.999 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:52.000 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:52.000 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.001 INFO:tasks.ceph.osd.4.magna078.stderr:   -41> 2015-08-12 20:37:51.961266 7f34899e1700 -1 *** Caught signal (Aborted) **
2015-08-12T20:37:52.002 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:52.002 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.003 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:52.044 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:52.045 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:52.085 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:52.086 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:52.126 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:52.127 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:52.167 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:52.168 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:52.208 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:52.209 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:52.249 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:52.250 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:52.290 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:52.291 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:52.331 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:52.332 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:52.333 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:52.334 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:52.335 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:52.335 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.593 INFO:tasks.ceph.osd.4.magna078.stderr:daemon-helper: command crashed with signal 6
2015-08-12T20:37:52.939 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.4 to 1 (8655362)
2015-08-12T20:37:53.296 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1'
2015-08-12T20:37:55.333 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.5 to 1 (8655362)
2015-08-12T20:37:55.342 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1'
2015-08-12T20:37:56.806 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.3 to 1 (8655362)
2015-08-12T20:37:56.814 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete
2015-08-12T20:37:56.815 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2015-08-12T20:37:57.090 INFO:teuthology.orchestra.run.magna087.stderr:dumped all in format json
2015-08-12T20:37:57.098 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2015-08-12T20:37:57.434 INFO:teuthology.orchestra.run.magna087.stderr:dumped all in format json
2015-08-12T20:37:57.441 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty'

Full logs

2015-08-12T20:28:44.187 INFO:teuthology.orchestra.run.magna078.stdout: marking collection for removal
2015-08-12T20:28:44.188 INFO:teuthology.orchestra.run.magna078.stdout:setting '_remove' omap key
2015-08-12T20:28:44.195 INFO:teuthology.orchestra.run.magna078.stdout:finish_remove_pgs removing 0.2_head pgid is 0.2
2015-08-12T20:28:44.196 INFO:teuthology.orchestra.run.magna078.stdout:remove_coll 0.2_head
2015-08-12T20:28:44.201 INFO:teuthology.orchestra.run.magna078.stdout:remove 2//head//0
2015-08-12T20:28:44.202 INFO:teuthology.orchestra.run.magna078.stdout:remove a05b9102/rb.0.1010.238e1f29.000000000848/head//0
2015-08-12T20:28:44.203 INFO:teuthology.orchestra.run.magna078.stdout:remove b5fe2302/rb.0.1010.238e1f29.0000000008bc/head//0
2015-08-12T20:28:44.204 INFO:teuthology.orchestra.run.magna078.stdout:remove 8131a302/rb.0.1010.238e1f29.0000000005d3/head//0
2015-08-12T20:28:44.205 INFO:teuthology.orchestra.run.magna078.stdout:remove b325a402/rb.0.1010.238e1f29.0000000003f8/head//0
2015-08-12T20:28:44.206 INFO:teuthology.orchestra.run.magna078.stdout:remove 97924602/rb.0.1010.238e1f29.00000000005e/head//0
2015-08-12T20:28:44.206 INFO:teuthology.orchestra.run.magna078.stdout:remove 7bcba602/rb.0.1010.238e1f29.00000000051e/head//0
2015-08-12T20:28:44.207 INFO:teuthology.orchestra.run.magna078.stdout:remove b2fda602/rb.0.1010.238e1f29.0000000006c6/head//0
2015-08-12T20:28:44.208 INFO:teuthology.orchestra.run.magna078.stdout:remove c053a702/rb.0.1010.238e1f29.000000000543/head//0
2015-08-12T20:28:44.209 INFO:teuthology.orchestra.run.magna078.stdout:remove f58c3802/rb.0.1010.238e1f29.000000000acd/head//0
2015-08-12T20:28:44.209 INFO:teuthology.orchestra.run.magna078.stdout:remove 96472b02/rb.0.1010.238e1f29.000000000171/head//0
2015-08-12T20:28:44.210 INFO:teuthology.orchestra.run.magna078.stdout:remove c120fc02/rb.0.1010.238e1f29.0000000005a5/head//0
2015-08-12T20:28:44.210 INFO:teuthology.orchestra.run.magna078.stdout:remove 2284ae02/rb.0.1010.238e1f29.000000000691/head//0
2015-08-12T20:28:44.211 INFO:teuthology.orchestra.run.magna078.stdout:remove db7a1f02/rb.0.1010.238e1f29.0000000005fd/head//0
2015-08-12T20:28:44.212 INFO:teuthology.orchestra.run.magna078.stdout:remove 5cba1f02/rb.0.1010.238e1f29.000000000a4e/head//0
2015-08-12T20:28:44.212 INFO:teuthology.orchestra.run.magna078.stdout:remove 5b15bf02/rb.0.1010.238e1f29.000000000087/head//0
2015-08-12T20:28:44.213 INFO:teuthology.orchestra.run.magna078.stdout:remove e8874222/rb.0.1010.238e1f29.000000000c27/head//0
2015-08-12T20:28:44.213 INFO:teuthology.orchestra.run.magna078.stdout:remove f5e15222/rb.0.1010.238e1f29.000000000786/head//0
2015-08-12T20:28:44.214 INFO:teuthology.orchestra.run.magna078.stdout:remove 6b6e1722/rb.0.1010.238e1f29.000000000cd9/head//0
2015-08-12T20:28:44.214 INFO:teuthology.orchestra.run.magna078.stdout:remove a6ea5722/rb.0.1010.238e1f29.00000000076e/head//0
2015-08-12T20:28:44.215 INFO:teuthology.orchestra.run.magna078.stdout:remove fa77a922/rb.0.1010.238e1f29.000000000141/head//0
2015-08-12T20:28:44.215 INFO:teuthology.orchestra.run.magna078.stdout:remove 4f2bb922/rb.0.1010.238e1f29.000000000262/head//0
2015-08-12T20:28:44.216 INFO:teuthology.orchestra.run.magna078.stdout:remove 36435b22/rb.0.1010.238e1f29.000000000d0c/head//0
2015-08-12T20:28:44.217 INFO:teuthology.orchestra.run.magna078.stdout:remove f071bb22/rb.0.1010.238e1f29.0000000001a4/head//0
2015-08-12T20:28:44.217 INFO:teuthology.orchestra.run.magna078.stdout:remove bba6142/rb.0.1010.238e1f29.00000000029c/head//0
2015-08-12T20:28:44.218 INFO:teuthology.orchestra.run.magna078.stdout:remove efd87142/rb.0.1010.238e1f29.000000000605/head//0
2015-08-12T20:28:44.219 INFO:teuthology.orchestra.run.magna078.stdout:remove 295db142/rb.0.1010.238e1f29.000000000105/head//0
2015-08-12T20:28:44.219 INFO:teuthology.orchestra.run.magna078.stdout:remove bbc48242/rb.0.1010.238e1f29.000000000bb7/head//0
2015-08-12T20:28:44.220 INFO:teuthology.orchestra.run.magna078.stdout:remove bafbb242/rb.0.1010.238e1f29.00000000094c/head//0
2015-08-12T20:28:44.220 INFO:teuthology.orchestra.run.magna078.stdout:remove 9e98d242/rb.0.1010.238e1f29.000000000724/head//0
2015-08-12T20:28:44.221 INFO:teuthology.orchestra.run.magna078.stdout:remove 40f97342/rb.0.1010.238e1f29.000000000806/head//0
2015-08-12T20:28:44.237 INFO:teuthology.orchestra.run.magna078.stdout:remove 7d46e342/rb.0.1010.238e1f29.0000000001a6/head//0
2015-08-12T20:28:44.239 INFO:teuthology.orchestra.run.magna078.stdout:remove aaa5a642/rb.0.1010.238e1f29.000000000dab/head//0
2015-08-12T20:28:44.240 INFO:teuthology.orchestra.run.magna078.stdout:remove 6e1f642/rb.0.1010.238e1f29.00000000020e/head//0
2015-08-12T20:28:44.241 INFO:teuthology.orchestra.run.magna078.stdout:remove 9b042742/rb.0.1010.238e1f29.0000000004b2/head//0
2015-08-12T20:28:44.241 INFO:teuthology.orchestra.run.magna078.stdout:remove 9584e842/rb.0.1010.238e1f29.000000000c17/head//0
2015-08-12T20:28:44.242 INFO:teuthology.orchestra.run.magna078.stdout:remove fe17c942/rb.0.1010.238e1f29.0000000002d2/head//0
2015-08-12T20:28:44.243 INFO:teuthology.orchestra.run.magna078.stdout:remove 5fdd4b42/rb.0.1010.238e1f29.0000000008ff/head//0
2015-08-12T20:28:44.244 INFO:teuthology.orchestra.run.magna078.stdout:remove 7f939b42/rb.0.1010.238e1f29.00000000004e/head//0
2015-08-12T20:28:44.245 INFO:teuthology.orchestra.run.magna078.stdout:remove 31166c42/rb.0.1010.238e1f29.000000000c69/head//0
2015-08-12T20:28:44.246 INFO:teuthology.orchestra.run.magna078.stdout:remove 7988cf42/rb.0.1010.238e1f29.0000000008fe/head//0
2015-08-12T20:28:44.246 INFO:teuthology.orchestra.run.magna078.stdout:remove df1d8162/rb.0.1010.238e1f29.00000000019b/head//0
2015-08-12T20:28:44.247 INFO:teuthology.orchestra.run.magna078.stdout:remove b6a2d162/rb.0.1010.238e1f29.000000000a2f/head//0
2015-08-12T20:28:44.248 INFO:teuthology.orchestra.run.magna078.stdout:remove 9ad44262/rb.0.1010.238e1f29.0000000005ed/head//0
2015-08-12T20:28:44.249 INFO:teuthology.orchestra.run.magna078.stdout:remove 9a55a262/rb.0.1010.238e1f29.00000000002c/head//0
2015-08-12T20:28:44.249 INFO:teuthology.orchestra.run.magna078.stdout:remove 5e741462/rb.0.1010.238e1f29.000000000b17/head//0
2015-08-12T20:28:44.250 INFO:teuthology.orchestra.run.magna078.stdout:remove fa1c6462/rb.0.1010.238e1f29.000000000197/head//0
2015-08-12T20:28:44.251 INFO:teuthology.orchestra.run.magna078.stdout:remove c5523562/rb.0.1010.238e1f29.000000000b77/head//0
2015-08-12T20:28:44.252 INFO:teuthology.orchestra.run.magna078.stdout:remove aa8be562/rb.0.1010.238e1f29.00000000039c/head//0
2015-08-12T20:28:44.252 INFO:teuthology.orchestra.run.magna078.stdout:remove 91659662/rb.0.1010.238e1f29.000000000b0c/head//0
2015-08-12T20:28:44.253 INFO:teuthology.orchestra.run.magna078.stdout:remove 1294d762/rb.0.1010.238e1f29.000000000505/head//0
2015-08-12T20:28:44.254 INFO:teuthology.orchestra.run.magna078.stdout:remove 1ac4bb62/rb.0.1010.238e1f29.0000000005d9/head//0
2015-08-12T20:28:44.255 INFO:teuthology.orchestra.run.magna078.stdout:remove 4fc2cb62/rb.0.1010.238e1f29.000000000563/head//0
2015-08-12T20:28:44.256 INFO:teuthology.orchestra.run.magna078.stdout:remove ab545d62/rb.0.1010.238e1f29.0000000001f9/head//0
2015-08-12T20:28:44.257 INFO:teuthology.orchestra.run.magna078.stdout:remove 6570ed62/rb.0.1010.238e1f29.000000000717/head//0
2015-08-12T20:28:44.257 INFO:teuthology.orchestra.run.magna078.stdout:remove de936e62/rb.0.1010.238e1f29.000000000668/head//0
2015-08-12T20:28:44.258 INFO:teuthology.orchestra.run.magna078.stdout:remove 46dbe62/rb.0.1010.238e1f29.000000000733/head//0
2015-08-12T20:28:44.259 INFO:teuthology.orchestra.run.magna078.stdout:remove 63954182/rb.0.1010.238e1f29.000000000607/head//0
2015-08-12T20:28:44.260 INFO:teuthology.orchestra.run.magna078.stdout:remove 61d8b282/rb.0.1010.238e1f29.000000000b5c/head//0
2015-08-12T20:28:44.261 INFO:teuthology.orchestra.run.magna078.stdout:remove 10fb7482/rb.0.1010.238e1f29.000000000f7d/head//0
2015-08-12T20:28:44.261 INFO:teuthology.orchestra.run.magna078.stdout:remove f497e482/rb.0.1010.238e1f29.000000000a15/head//0
2015-08-12T20:28:44.264 INFO:teuthology.orchestra.run.magna078.stdout:remove 9f08f482/rb.0.1010.238e1f29.000000000ba6/head//0
2015-08-12T20:28:44.265 INFO:teuthology.orchestra.run.magna078.stdout:remove af961582/rb.0.1010.238e1f29.000000000291/head//0
2015-08-12T20:28:44.266 INFO:teuthology.orchestra.run.magna078.stdout:remove fb3c3782/rb.0.1010.238e1f29.000000000239/head//0
2015-08-12T20:28:44.267 INFO:teuthology.orchestra.run.magna078.stdout:remove 98a95882/rb.0.1010.238e1f29.000000000b08/head//0
2015-08-12T20:28:44.268 INFO:teuthology.orchestra.run.magna078.stdout:remove 1e0a5982/rb.0.1010.238e1f29.00000000033e/head//0
2015-08-12T20:28:44.268 INFO:teuthology.orchestra.run.magna078.stdout:remove 111bca82/rb.0.1010.238e1f29.0000000005ef/head//0
2015-08-12T20:28:44.269 INFO:teuthology.orchestra.run.magna078.stdout:remove 9b681d82/rb.0.1010.238e1f29.000000000449/head//0
2015-08-12T20:28:44.270 INFO:teuthology.orchestra.run.magna078.stdout:remove 568ebd82/rb.0.1010.238e1f29.000000000567/head//0
2015-08-12T20:28:44.271 INFO:teuthology.orchestra.run.magna078.stdout:remove bf224e82/rb.0.1010.238e1f29.000000000a8d/head//0
2015-08-12T20:28:44.271 INFO:teuthology.orchestra.run.magna078.stdout:remove a92b3f82/rb.0.1010.238e1f29.000000000b5f/head//0
2015-08-12T20:28:44.274 INFO:teuthology.orchestra.run.magna078.stdout:remove f0ef9f82/rb.0.1010.238e1f29.0000000000cb/head//0
2015-08-12T20:28:44.275 INFO:teuthology.orchestra.run.magna078.stdout:remove 3f17e0a2/rb.0.1010.238e1f29.00000000080a/head//0
2015-08-12T20:28:44.276 INFO:teuthology.orchestra.run.magna078.stdout:remove f05791a2/rb.0.1010.238e1f29.0000000001d2/head//0
2015-08-12T20:28:44.276 INFO:teuthology.orchestra.run.magna078.stdout:remove 3364b1a2/rb.0.1010.238e1f29.0000000008fc/head//0
2015-08-12T20:28:44.277 INFO:teuthology.orchestra.run.magna078.stdout:remove 984732a2/rb.0.1010.238e1f29.000000000dc8/head//0
2015-08-12T20:28:44.278 INFO:teuthology.orchestra.run.magna078.stdout:remove 3aa4e2a2/rb.0.1010.238e1f29.000000000aa0/head//0
2015-08-12T20:28:44.279 INFO:teuthology.orchestra.run.magna078.stdout:remove 7d1af2a2/rb.0.1010.238e1f29.00000000017f/head//0
2015-08-12T20:28:44.279 INFO:teuthology.orchestra.run.magna078.stdout:remove 856fd4a2/rb.0.1010.238e1f29.000000000b1e/head//0
2015-08-12T20:28:44.280 INFO:teuthology.orchestra.run.magna078.stdout:remove af4b45a2/rb.0.1010.238e1f29.0000000008a8/head//0
2015-08-12T20:28:44.281 INFO:teuthology.orchestra.run.magna078.stdout:remove cd9be5a2/rb.0.1010.238e1f29.0000000005dc/head//0
2015-08-12T20:28:44.282 INFO:teuthology.orchestra.run.magna078.stdout:remove 76d406a2/rb.0.1010.238e1f29.000000000a45/head//0
2015-08-12T20:28:44.282 INFO:teuthology.orchestra.run.magna078.stdout:remove a9a6a8a2/rb.0.1010.238e1f29.00000000038c/head//0
2015-08-12T20:28:44.283 INFO:teuthology.orchestra.run.magna078.stdout:remove d6a819a2/rb.0.1010.238e1f29.000000000d90/head//0
2015-08-12T20:28:44.284 INFO:teuthology.orchestra.run.magna078.stdout:remove 7cf3bea2/rb.0.1010.238e1f29.00000000066b/head//0
2015-08-12T20:28:44.284 INFO:teuthology.orchestra.run.magna078.stdout:remove a4478fa2/rb.0.1010.238e1f29.0000000000a1/head//0
2015-08-12T20:28:44.285 INFO:teuthology.orchestra.run.magna078.stdout:remove 6cc0c0c2/rb.0.1010.238e1f29.0000000004b3/head//0
2015-08-12T20:28:44.286 INFO:teuthology.orchestra.run.magna078.stdout:remove e78ad1c2/rb.0.1010.238e1f29.00000000085e/head//0
2015-08-12T20:28:44.287 INFO:teuthology.orchestra.run.magna078.stdout:remove fc3b43c2/rb.0.1010.238e1f29.000000000cc5/head//0
2015-08-12T20:28:44.288 INFO:teuthology.orchestra.run.magna078.stdout:remove 43d7f5c2/rb.0.1010.238e1f29.00000000029f/head//0
2015-08-12T20:28:44.289 INFO:teuthology.orchestra.run.magna078.stdout:remove 98b8b6c2/rb.0.1010.238e1f29.000000000007/head//0
2015-08-12T20:28:44.294 INFO:teuthology.orchestra.run.magna078.stdout:remove 839427c2/rb.0.1010.238e1f29.000000000c2e/head//0
2015-08-12T20:28:44.294 INFO:teuthology.orchestra.run.magna078.stdout:remove 7cfbe7c2/rb.0.1010.238e1f29.000000000777/head//0
2015-08-12T20:28:44.295 INFO:teuthology.orchestra.run.magna078.stdout:remove 81f059c2/rb.0.1010.238e1f29.000000000486/head//0
2015-08-12T20:28:44.296 INFO:teuthology.orchestra.run.magna078.stdout:remove 62babc2/rb.0.1010.238e1f29.000000000b14/head//0
2015-08-12T20:28:44.297 INFO:teuthology.orchestra.run.magna078.stdout:remove 69ad1cc2/rb.0.1010.238e1f29.000000000497/head//0
2015-08-12T20:28:44.298 INFO:teuthology.orchestra.run.magna078.stdout:remove 858b0dc2/rb.0.1010.238e1f29.00000000047a/head//0
2015-08-12T20:28:44.298 INFO:teuthology.orchestra.run.magna078.stdout:remove 81028dc2/rb.0.1010.238e1f29.0000000008d6/head//0
2015-08-12T20:28:44.299 INFO:teuthology.orchestra.run.magna078.stdout:remove beaa8dc2/rb.0.1010.238e1f29.00000000031e/head//0
2015-08-12T20:28:44.300 INFO:teuthology.orchestra.run.magna078.stdout:remove eba2fc2/rb.0.1010.238e1f29.0000000008d7/head//0
2015-08-12T20:28:44.300 INFO:teuthology.orchestra.run.magna078.stdout:remove 7a3391e2/rb.0.1010.238e1f29.000000000742/head//0
2015-08-12T20:28:44.301 INFO:teuthology.orchestra.run.magna078.stdout:remove 623ec1e2/rb.0.1010.238e1f29.000000000460/head//0
2015-08-12T20:28:44.304 INFO:teuthology.orchestra.run.magna078.stdout:remove 2d5e43e2/rb.0.1010.238e1f29.0000000003dd/head//0
2015-08-12T20:28:44.305 INFO:teuthology.orchestra.run.magna078.stdout:remove 60d6d3e2/rb.0.1010.238e1f29.0000000009b7/head//0
2015-08-12T20:28:44.305 INFO:teuthology.orchestra.run.magna078.stdout:remove 895c04e2/rb.0.1010.238e1f29.000000000bee/head//0
2015-08-12T20:28:44.306 INFO:teuthology.orchestra.run.magna078.stdout:remove 8e9844e2/rb.0.1010.238e1f29.0000000007b1/head//0
2015-08-12T20:28:44.307 INFO:teuthology.orchestra.run.magna078.stdout:remove f290e6e2/rb.0.1010.238e1f29.00000000012e/head//0
2015-08-12T20:28:44.308 INFO:teuthology.orchestra.run.magna078.stdout:remove 80127e2/rb.0.1010.238e1f29.0000000005ae/head//0
2015-08-12T20:28:44.308 INFO:teuthology.orchestra.run.magna078.stdout:remove c92e67e2/rb.0.1010.238e1f29.0000000003f5/head//0
2015-08-12T20:28:44.309 INFO:teuthology.orchestra.run.magna078.stdout:remove e0e6e7e2/rb.0.1010.238e1f29.00000000009c/head//0
2015-08-12T20:28:44.310 INFO:teuthology.orchestra.run.magna078.stdout:remove e31638e2/rb.0.1010.238e1f29.000000000d9e/head//0
2015-08-12T20:28:44.310 INFO:teuthology.orchestra.run.magna078.stdout:remove 2e390be2/rb.0.1010.238e1f29.000000000b36/head//0
2015-08-12T20:28:44.311 INFO:teuthology.orchestra.run.magna078.stdout:remove 74d93de2/rb.0.1010.238e1f29.000000000698/head//0
2015-08-12T20:28:44.312 INFO:teuthology.orchestra.run.magna078.stdout:remove a4091ee2/rb.0.1010.238e1f29.000000000199/head//0
2015-08-12T20:28:44.312 INFO:teuthology.orchestra.run.magna078.stdout:remove b82e6ee2/rb.0.1010.238e1f29.0000000013bf/head//0
2015-08-12T20:28:44.313 INFO:teuthology.orchestra.run.magna078.stdout:remove 5311efe2/rb.0.1010.238e1f29.000000000596/head//0
2015-08-12T20:28:44.323 INFO:teuthology.orchestra.run.magna078.stdout:Remove successful
2015-08-12T20:28:44.411 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op import --file /home/ubuntu/cephtest/data/exp.0.2.3'
2015-08-12T20:28:44.779 INFO:teuthology.orchestra.run.magna078.stdout:Importing pgid 0.2
2015-08-12T20:28:44.784 INFO:teuthology.orchestra.run.magna078.stdout:Write a05b9102/rb.0.1010.238e1f29.000000000848/head//0
2015-08-12T20:28:44.836 INFO:teuthology.orchestra.run.magna078.stdout:Write b5fe2302/rb.0.1010.238e1f29.0000000008bc/head//0
2015-08-12T20:28:44.883 INFO:teuthology.orchestra.run.magna078.stdout:Write 8131a302/rb.0.1010.238e1f29.0000000005d3/head//0
2015-08-12T20:28:44.921 INFO:teuthology.orchestra.run.magna078.stdout:Write b325a402/rb.0.1010.238e1f29.0000000003f8/head//0
2015-08-12T20:28:44.967 INFO:teuthology.orchestra.run.magna078.stdout:Write 97924602/rb.0.1010.238e1f29.00000000005e/head//0
2015-08-12T20:28:45.013 INFO:teuthology.orchestra.run.magna078.stdout:Write 7bcba602/rb.0.1010.238e1f29.00000000051e/head//0
2015-08-12T20:28:45.059 INFO:teuthology.orchestra.run.magna078.stdout:Write b2fda602/rb.0.1010.238e1f29.0000000006c6/head//0
2015-08-12T20:28:45.098 INFO:teuthology.orchestra.run.magna078.stdout:Write c053a702/rb.0.1010.238e1f29.000000000543/head//0
2015-08-12T20:28:45.144 INFO:teuthology.orchestra.run.magna078.stdout:Write f58c3802/rb.0.1010.238e1f29.000000000acd/head//0
2015-08-12T20:28:45.182 INFO:teuthology.orchestra.run.magna078.stdout:Write 96472b02/rb.0.1010.238e1f29.000000000171/head//0
2015-08-12T20:28:45.228 INFO:teuthology.orchestra.run.magna078.stdout:Write c120fc02/rb.0.1010.238e1f29.0000000005a5/head//0
2015-08-12T20:28:45.357 INFO:teuthology.orchestra.run.magna078.stdout:Write 2284ae02/rb.0.1010.238e1f29.000000000691/head//0
2015-08-12T20:28:45.463 INFO:teuthology.orchestra.run.magna078.stdout:Write db7a1f02/rb.0.1010.238e1f29.0000000005fd/head//0
2015-08-12T20:28:45.568 INFO:teuthology.orchestra.run.magna078.stdout:Write 5cba1f02/rb.0.1010.238e1f29.000000000a4e/head//0
2015-08-12T20:28:45.809 INFO:teuthology.orchestra.run.magna078.stdout:Write 5b15bf02/rb.0.1010.238e1f29.000000000087/head//0
2015-08-12T20:28:45.891 INFO:teuthology.orchestra.run.magna078.stdout:Write e8874222/rb.0.1010.238e1f29.000000000c27/head//0
2015-08-12T20:28:45.994 INFO:teuthology.orchestra.run.magna078.stdout:Write f5e15222/rb.0.1010.238e1f29.000000000786/head//0
2015-08-12T20:28:46.110 INFO:teuthology.orchestra.run.magna078.stdout:Write 6b6e1722/rb.0.1010.238e1f29.000000000cd9/head//0
2015-08-12T20:28:46.182 INFO:teuthology.orchestra.run.magna078.stdout:Write a6ea5722/rb.0.1010.238e1f29.00000000076e/head//0
2015-08-12T20:28:46.302 INFO:teuthology.orchestra.run.magna078.stdout:Write fa77a922/rb.0.1010.238e1f29.000000000141/head//0
2015-08-12T20:28:46.391 INFO:teuthology.orchestra.run.magna078.stdout:Write 4f2bb922/rb.0.1010.238e1f29.000000000262/head//0
2015-08-12T20:28:46.497 INFO:teuthology.orchestra.run.magna078.stdout:Write 36435b22/rb.0.1010.238e1f29.000000000d0c/head//0
2015-08-12T20:28:46.601 INFO:teuthology.orchestra.run.magna078.stdout:Write f071bb22/rb.0.1010.238e1f29.0000000001a4/head//0
2015-08-12T20:28:46.664 INFO:teuthology.orchestra.run.magna078.stdout:Write bba6142/rb.0.1010.238e1f29.00000000029c/head//0
2015-08-12T20:28:46.800 INFO:teuthology.orchestra.run.magna078.stdout:Write efd87142/rb.0.1010.238e1f29.000000000605/head//0
2015-08-12T20:28:46.850 INFO:teuthology.orchestra.run.magna078.stdout:Write 295db142/rb.0.1010.238e1f29.000000000105/head//0
2015-08-12T20:28:46.888 INFO:teuthology.orchestra.run.magna078.stdout:Write bbc48242/rb.0.1010.238e1f29.000000000bb7/head//0
2015-08-12T20:28:47.091 INFO:teuthology.orchestra.run.magna078.stdout:Write bafbb242/rb.0.1010.238e1f29.00000000094c/head//0
2015-08-12T20:28:47.315 INFO:teuthology.orchestra.run.magna078.stdout:Write 9e98d242/rb.0.1010.238e1f29.000000000724/head//0
2015-08-12T20:28:47.551 INFO:teuthology.orchestra.run.magna078.stdout:Write 40f97342/rb.0.1010.238e1f29.000000000806/head//0
2015-08-12T20:28:47.691 INFO:teuthology.orchestra.run.magna078.stdout:Write 7d46e342/rb.0.1010.238e1f29.0000000001a6/head//0
2015-08-12T20:28:47.751 INFO:teuthology.orchestra.run.magna078.stdout:Write aaa5a642/rb.0.1010.238e1f29.000000000dab/head//0
2015-08-12T20:28:48.283 INFO:teuthology.orchestra.run.magna078.stdout:Write 6e1f642/rb.0.1010.238e1f29.00000000020e/head//0
2015-08-12T20:28:48.323 INFO:teuthology.orchestra.run.magna078.stdout:Write 9b042742/rb.0.1010.238e1f29.0000000004b2/head//0
2015-08-12T20:28:48.324 INFO:teuthology.orchestra.run.magna078.stdout:Write 9584e842/rb.0.1010.238e1f29.000000000c17/head//0
2015-08-12T20:28:48.324 INFO:teuthology.orchestra.run.magna078.stdout:Write fe17c942/rb.0.1010.238e1f29.0000000002d2/head//0
2015-08-12T20:28:48.325 INFO:teuthology.orchestra.run.magna078.stdout:Write 5fdd4b42/rb.0.1010.238e1f29.0000000008ff/head//0
2015-08-12T20:28:48.326 INFO:teuthology.orchestra.run.magna078.stdout:Write 7f939b42/rb.0.1010.238e1f29.00000000004e/head//0
2015-08-12T20:28:48.327 INFO:teuthology.orchestra.run.magna078.stdout:Write 31166c42/rb.0.1010.238e1f29.000000000c69/head//0
2015-08-12T20:28:48.327 INFO:teuthology.orchestra.run.magna078.stdout:Write 7988cf42/rb.0.1010.238e1f29.0000000008fe/head//0
2015-08-12T20:28:48.378 INFO:teuthology.orchestra.run.magna078.stdout:Write df1d8162/rb.0.1010.238e1f29.00000000019b/head//0
2015-08-12T20:28:48.716 INFO:teuthology.orchestra.run.magna078.stdout:Write b6a2d162/rb.0.1010.238e1f29.000000000a2f/head//0
2015-08-12T20:28:48.850 INFO:teuthology.orchestra.run.magna078.stdout:Write 9ad44262/rb.0.1010.238e1f29.0000000005ed/head//0
2015-08-12T20:28:48.925 INFO:teuthology.orchestra.run.magna078.stdout:Write 9a55a262/rb.0.1010.238e1f29.00000000002c/head//0
2015-08-12T20:28:48.971 INFO:teuthology.orchestra.run.magna078.stdout:Write 5e741462/rb.0.1010.238e1f29.000000000b17/head//0
2015-08-12T20:28:49.010 INFO:teuthology.orchestra.run.magna078.stdout:Write fa1c6462/rb.0.1010.238e1f29.000000000197/head//0
2015-08-12T20:28:49.065 INFO:teuthology.orchestra.run.magna078.stdout:Write c5523562/rb.0.1010.238e1f29.000000000b77/head//0
2015-08-12T20:28:49.111 INFO:teuthology.orchestra.run.magna078.stdout:Write aa8be562/rb.0.1010.238e1f29.00000000039c/head//0
2015-08-12T20:28:49.144 INFO:teuthology.orchestra.run.magna078.stdout:Write 91659662/rb.0.1010.238e1f29.000000000b0c/head//0
2015-08-12T20:28:49.200 INFO:teuthology.orchestra.run.magna078.stdout:Write 1294d762/rb.0.1010.238e1f29.000000000505/head//0
2015-08-12T20:28:49.247 INFO:teuthology.orchestra.run.magna078.stdout:Write 1ac4bb62/rb.0.1010.238e1f29.0000000005d9/head//0
2015-08-12T20:28:49.293 INFO:teuthology.orchestra.run.magna078.stdout:Write 4fc2cb62/rb.0.1010.238e1f29.000000000563/head//0
2015-08-12T20:28:49.331 INFO:teuthology.orchestra.run.magna078.stdout:Write ab545d62/rb.0.1010.238e1f29.0000000001f9/head//0
2015-08-12T20:28:49.687 INFO:teuthology.orchestra.run.magna078.stdout:Write 6570ed62/rb.0.1010.238e1f29.000000000717/head//0
2015-08-12T20:28:49.908 INFO:teuthology.orchestra.run.magna078.stdout:Write de936e62/rb.0.1010.238e1f29.000000000668/head//0
2015-08-12T20:28:49.987 INFO:teuthology.orchestra.run.magna078.stdout:Write 46dbe62/rb.0.1010.238e1f29.000000000733/head//0
2015-08-12T20:28:50.059 INFO:teuthology.orchestra.run.magna078.stdout:Write 63954182/rb.0.1010.238e1f29.000000000607/head//0
2015-08-12T20:28:50.149 INFO:teuthology.orchestra.run.magna078.stdout:Write 61d8b282/rb.0.1010.238e1f29.000000000b5c/head//0
2015-08-12T20:28:50.236 INFO:teuthology.orchestra.run.magna078.stdout:Write 10fb7482/rb.0.1010.238e1f29.000000000f7d/head//0
2015-08-12T20:28:50.266 INFO:teuthology.orchestra.run.magna078.stdout:Write f497e482/rb.0.1010.238e1f29.000000000a15/head//0
2015-08-12T20:28:50.379 INFO:teuthology.orchestra.run.magna078.stdout:Write 9f08f482/rb.0.1010.238e1f29.000000000ba6/head//0
2015-08-12T20:28:50.460 INFO:teuthology.orchestra.run.magna078.stdout:Write af961582/rb.0.1010.238e1f29.000000000291/head//0
2015-08-12T20:28:50.540 INFO:teuthology.orchestra.run.magna078.stdout:Write fb3c3782/rb.0.1010.238e1f29.000000000239/head//0
2015-08-12T20:28:50.587 INFO:teuthology.orchestra.run.magna078.stdout:Write 98a95882/rb.0.1010.238e1f29.000000000b08/head//0
2015-08-12T20:28:50.634 INFO:teuthology.orchestra.run.magna078.stdout:Write 1e0a5982/rb.0.1010.238e1f29.00000000033e/head//0
2015-08-12T20:28:50.680 INFO:teuthology.orchestra.run.magna078.stdout:Write 111bca82/rb.0.1010.238e1f29.0000000005ef/head//0
2015-08-12T20:28:50.718 INFO:teuthology.orchestra.run.magna078.stdout:Write 9b681d82/rb.0.1010.238e1f29.000000000449/head//0
2015-08-12T20:28:50.864 INFO:teuthology.orchestra.run.magna078.stdout:Write 568ebd82/rb.0.1010.238e1f29.000000000567/head//0
2015-08-12T20:28:51.015 INFO:teuthology.orchestra.run.magna078.stdout:Write bf224e82/rb.0.1010.238e1f29.000000000a8d/head//0
2015-08-12T20:28:51.198 INFO:teuthology.orchestra.run.magna078.stdout:Write a92b3f82/rb.0.1010.238e1f29.000000000b5f/head//0
2015-08-12T20:28:51.398 INFO:teuthology.orchestra.run.magna078.stdout:Write f0ef9f82/rb.0.1010.238e1f29.0000000000cb/head//0
2015-08-12T20:28:51.547 INFO:teuthology.orchestra.run.magna078.stdout:Write 3f17e0a2/rb.0.1010.238e1f29.00000000080a/head//0
2015-08-12T20:28:51.609 INFO:teuthology.orchestra.run.magna078.stdout:Write f05791a2/rb.0.1010.238e1f29.0000000001d2/head//0
2015-08-12T20:28:51.683 INFO:teuthology.orchestra.run.magna078.stdout:Write 3364b1a2/rb.0.1010.238e1f29.0000000008fc/head//0
2015-08-12T20:28:51.730 INFO:teuthology.orchestra.run.magna078.stdout:Write 984732a2/rb.0.1010.238e1f29.000000000dc8/head//0
2015-08-12T20:28:51.784 INFO:teuthology.orchestra.run.magna078.stdout:Write 3aa4e2a2/rb.0.1010.238e1f29.000000000aa0/head//0
2015-08-12T20:28:51.820 INFO:teuthology.orchestra.run.magna078.stdout:Write 7d1af2a2/rb.0.1010.238e1f29.00000000017f/head//0
2015-08-12T20:28:51.858 INFO:teuthology.orchestra.run.magna078.stdout:Write 856fd4a2/rb.0.1010.238e1f29.000000000b1e/head//0
2015-08-12T20:28:51.896 INFO:teuthology.orchestra.run.magna078.stdout:Write af4b45a2/rb.0.1010.238e1f29.0000000008a8/head//0
2015-08-12T20:28:51.934 INFO:teuthology.orchestra.run.magna078.stdout:Write cd9be5a2/rb.0.1010.238e1f29.0000000005dc/head//0
2015-08-12T20:28:52.209 INFO:teuthology.orchestra.run.magna078.stdout:Write 76d406a2/rb.0.1010.238e1f29.000000000a45/head//0
2015-08-12T20:28:52.492 INFO:teuthology.orchestra.run.magna078.stdout:Write a9a6a8a2/rb.0.1010.238e1f29.00000000038c/head//0
2015-08-12T20:28:52.644 INFO:teuthology.orchestra.run.magna078.stdout:Write d6a819a2/rb.0.1010.238e1f29.000000000d90/head//0
2015-08-12T20:28:52.700 INFO:teuthology.orchestra.run.magna078.stdout:Write 7cf3bea2/rb.0.1010.238e1f29.00000000066b/head//0
2015-08-12T20:28:52.766 INFO:teuthology.orchestra.run.magna078.stdout:Write a4478fa2/rb.0.1010.238e1f29.0000000000a1/head//0
2015-08-12T20:28:53.283 INFO:teuthology.orchestra.run.magna078.stdout:Write 6cc0c0c2/rb.0.1010.238e1f29.0000000004b3/head//0
2015-08-12T20:28:53.322 INFO:teuthology.orchestra.run.magna078.stdout:Write e78ad1c2/rb.0.1010.238e1f29.00000000085e/head//0
2015-08-12T20:28:53.323 INFO:teuthology.orchestra.run.magna078.stdout:Write fc3b43c2/rb.0.1010.238e1f29.000000000cc5/head//0
2015-08-12T20:28:53.364 INFO:teuthology.orchestra.run.magna078.stdout:Write 43d7f5c2/rb.0.1010.238e1f29.00000000029f/head//0
2015-08-12T20:28:53.365 INFO:teuthology.orchestra.run.magna078.stdout:Write 98b8b6c2/rb.0.1010.238e1f29.000000000007/head//0
2015-08-12T20:28:53.406 INFO:teuthology.orchestra.run.magna078.stdout:Write 839427c2/rb.0.1010.238e1f29.000000000c2e/head//0
2015-08-12T20:28:53.407 INFO:teuthology.orchestra.run.magna078.stdout:Write 7cfbe7c2/rb.0.1010.238e1f29.000000000777/head//0
2015-08-12T20:28:53.447 INFO:teuthology.orchestra.run.magna078.stdout:Write 81f059c2/rb.0.1010.238e1f29.000000000486/head//0
2015-08-12T20:28:53.448 INFO:teuthology.orchestra.run.magna078.stdout:Write 62babc2/rb.0.1010.238e1f29.000000000b14/head//0
2015-08-12T20:28:53.489 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.462915 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.462914)
2015-08-12T20:28:53.583 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.575237 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.575237)
2015-08-12T20:28:53.666 INFO:teuthology.orchestra.run.magna078.stdout:Write 69ad1cc2/rb.0.1010.238e1f29.000000000497/head//0
2015-08-12T20:28:53.766 INFO:teuthology.orchestra.run.magna078.stdout:Write 858b0dc2/rb.0.1010.238e1f29.00000000047a/head//0
2015-08-12T20:28:53.837 INFO:teuthology.orchestra.run.magna078.stdout:Write 81028dc2/rb.0.1010.238e1f29.0000000008d6/head//0
2015-08-12T20:28:53.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:53.856988 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:33.856987)
2015-08-12T20:28:53.908 INFO:teuthology.orchestra.run.magna078.stdout:Write beaa8dc2/rb.0.1010.238e1f29.00000000031e/head//0
2015-08-12T20:28:53.971 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.963196 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.963195)
2015-08-12T20:28:54.036 INFO:teuthology.orchestra.run.magna078.stdout:Write eba2fc2/rb.0.1010.238e1f29.0000000008d7/head//0
2015-08-12T20:28:54.131 INFO:teuthology.orchestra.run.magna078.stdout:Write 7a3391e2/rb.0.1010.238e1f29.000000000742/head//0
2015-08-12T20:28:54.154 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:54.145905 7f2e489a7700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:34.145903)
2015-08-12T20:28:54.212 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:54.211687 7ff3458aa700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:34.211686)
2015-08-12T20:28:54.225 INFO:teuthology.orchestra.run.magna078.stdout:Write 623ec1e2/rb.0.1010.238e1f29.000000000460/head//0
2015-08-12T20:28:54.278 INFO:teuthology.orchestra.run.magna078.stdout:Write 2d5e43e2/rb.0.1010.238e1f29.0000000003dd/head//0
2015-08-12T20:28:54.324 INFO:teuthology.orchestra.run.magna078.stdout:Write 60d6d3e2/rb.0.1010.238e1f29.0000000009b7/head//0
2015-08-12T20:28:54.362 INFO:teuthology.orchestra.run.magna078.stdout:Write 895c04e2/rb.0.1010.238e1f29.000000000bee/head//0
2015-08-12T20:28:54.399 INFO:teuthology.orchestra.run.magna078.stdout:Write 8e9844e2/rb.0.1010.238e1f29.0000000007b1/head//0
2015-08-12T20:28:54.439 INFO:teuthology.orchestra.run.magna078.stdout:Write f290e6e2/rb.0.1010.238e1f29.00000000012e/head//0
2015-08-12T20:28:54.583 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:54.575467 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:34.575466)
2015-08-12T20:28:54.602 INFO:teuthology.orchestra.run.magna078.stdout:Write 80127e2/rb.0.1010.238e1f29.0000000005ae/head//0
2015-08-12T20:28:54.744 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:54.743771 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:34.743769)
2015-08-12T20:28:54.771 INFO:teuthology.orchestra.run.magna078.stdout:Write c92e67e2/rb.0.1010.238e1f29.0000000003f5/head//0
2015-08-12T20:28:54.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:54.857136 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:34.857135)
2015-08-12T20:28:54.986 INFO:teuthology.orchestra.run.magna078.stdout:Write e0e6e7e2/rb.0.1010.238e1f29.00000000009c/head//0
2015-08-12T20:28:55.130 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:55.129858 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:35.129856)
2015-08-12T20:28:55.200 INFO:teuthology.orchestra.run.magna078.stdout:Write e31638e2/rb.0.1010.238e1f29.000000000d9e/head//0
2015-08-12T20:28:55.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.203738 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.203737)
2015-08-12T20:28:55.338 INFO:teuthology.orchestra.run.magna078.stdout:Write 2e390be2/rb.0.1010.238e1f29.000000000b36/head//0
2015-08-12T20:28:55.408 INFO:teuthology.orchestra.run.magna078.stdout:Write 74d93de2/rb.0.1010.238e1f29.000000000698/head//0
2015-08-12T20:28:55.430 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.422616 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.422615)
2015-08-12T20:28:55.454 INFO:teuthology.orchestra.run.magna078.stdout:Write a4091ee2/rb.0.1010.238e1f29.000000000199/head//0
2015-08-12T20:28:55.500 INFO:teuthology.orchestra.run.magna078.stdout:Write b82e6ee2/rb.0.1010.238e1f29.0000000013bf/head//0
2015-08-12T20:28:55.523 INFO:teuthology.orchestra.run.magna078.stdout:Write 5311efe2/rb.0.1010.238e1f29.000000000596/head//0
2015-08-12T20:28:55.532 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:55.531059 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:35.531058)
2015-08-12T20:28:55.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:55.575630 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:35.575629)
2015-08-12T20:28:55.606 INFO:teuthology.orchestra.run.magna078.stdout:Import successful
2015-08-12T20:28:55.744 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:55.743939 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:35.743937)
2015-08-12T20:28:55.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:55.857233 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:35.857232)
2015-08-12T20:28:55.931 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.922930 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.922930)
2015-08-12T20:28:56.104 INFO:teuthology.orchestra.run.magna078:Running: 'rm -f /home/ubuntu/cephtest/data/exp.0.2.3'
2015-08-12T20:28:56.131 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:56.130115 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:36.130113)
2015-08-12T20:28:56.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:56.203887 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:36.203887)
2015-08-12T20:28:56.271 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:56.263494 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:36.263494)
2015-08-12T20:28:56.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:56.575772 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:36.575771)
2015-08-12T20:28:56.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:56.744048 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:36.744046)
2015-08-12T20:28:56.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:56.857381 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:36.857380)
2015-08-12T20:28:57.031 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:57.023248 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:37.023247)
2015-08-12T20:28:57.112 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:57.112016 7ff3458aa700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:37.112016)
2015-08-12T20:28:57.131 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:57.130278 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:37.130277)
2015-08-12T20:28:57.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:57.204051 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:37.204050)
2015-08-12T20:28:57.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:57.575918 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:37.575917)
2015-08-12T20:28:57.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:57.744158 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:37.744157)
2015-08-12T20:28:58.256 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:57.857483 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:37.857483)
2015-08-12T20:28:58.295 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:57.963853 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:37.963852)
2015-08-12T20:28:58.296 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:58.204288 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:38.204287)
2015-08-12T20:28:58.337 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:57.831342 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:37.831341)
2015-08-12T20:28:58.338 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:58.130474 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:38.130473)
2015-08-12T20:28:58.339 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:58.331587 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:38.331587)
2015-08-12T20:28:58.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:58.576018 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:38.576018)
2015-08-12T20:28:58.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:58.744337 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:38.744336)
2015-08-12T20:29:01.868 INFO:tasks.thrashosds.thrasher:in_osds:  [3, 2, 4, 1, 5, 0] out_osds:  [] dead_osds:  [3] live_osds:  [0, 2, 4, 1, 5]
2015-08-12T20:29:01.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-08-12T20:29:01.871 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [3, 2, 4, 1, 5, 0]
2015-08-12T20:29:01.872 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3'
2015-08-12T20:29:02.429 INFO:teuthology.orchestra.run.magna087.stderr:marked out osd.3.
2015-08-12T20:29:07.437 INFO:tasks.thrashosds.thrasher:in_osds:  [2, 4, 1, 5, 0] out_osds:  [3] dead_osds:  [3] live_osds:  [0, 2, 4, 1, 5]
2015-08-12T20:29:07.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-08-12T20:29:07.439 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 4, 1, 5]
2015-08-12T20:29:13.438 INFO:tasks.ceph.osd.2:Stopped
2015-08-12T20:29:13.439 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd
2015-08-12T20:29:13.440 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
2015-08-12T20:29:13.768 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 0.b --file /home/ubuntu/cephtest/data/exp.0.b.3'
2015-08-12T20:29:13.970 INFO:teuthology.orchestra.run.magna078.stderr:Exporting 0.b
2015-08-12T20:29:14.024 INFO:teuthology.orchestra.run.magna078.stderr:Read 3e47000b/rb.0.1010.238e1f29.0000000000ac/head//0
2015-08-12T20:29:14.072 INFO:teuthology.orchestra.run.magna078.stderr:Read b1bbb00b/rb.0.1010.238e1f29.000000000927/head//0
2015-08-12T20:29:14.077 INFO:teuthology.orchestra.run.magna078.stderr:Read d7f1f00b/rb.0.1010.238e1f29.000000000ac5/head//0
2015-08-12T20:29:14.081 INFO:teuthology.orchestra.run.magna078.stderr:Read b189f00b/rb.0.1010.238e1f29.0000000003bf/head//0
2015-08-12T20:29:14.085 INFO:teuthology.orchestra.run.magna078.stderr:Read b5e8510b/rb.0.1010.238e1f29.0000000003e9/head//0
2015-08-12T20:29:14.089 INFO:teuthology.orchestra.run.magna078.stderr:Read a8d2420b/rb.0.1010.238e1f29.000000000ccd/head//0
2015-08-12T20:29:14.093 INFO:teuthology.orchestra.run.magna078.stderr:Read 4cadd20b/rb.0.1010.238e1f29.00000000012a/head//0
2015-08-12T20:29:14.128 INFO:teuthology.orchestra.run.magna078.stderr:Read 15f8940b/rb.0.1010.238e1f29.000000000a67/head//0
2015-08-12T20:29:14.171 INFO:teuthology.orchestra.run.magna078.stderr:Read 32e9350b/rb.0.1010.238e1f29.00000000047b/head//0
2015-08-12T20:29:14.175 INFO:teuthology.orchestra.run.magna078.stderr:Read 47b850b/rb.0.1010.238e1f29.000000000692/head//0
2015-08-12T20:29:14.217 INFO:teuthology.orchestra.run.magna078.stderr:Read 2a57570b/rb.0.1010.238e1f29.00000000063d/head//0
2015-08-12T20:29:14.266 INFO:teuthology.orchestra.run.magna078.stderr:Read 908e80b/rb.0.1010.238e1f29.000000000325/head//0
2015-08-12T20:29:14.270 INFO:teuthology.orchestra.run.magna078.stderr:Read 4e17390b/rb.0.1010.238e1f29.000000000a41/head//0
2015-08-12T20:29:14.316 INFO:teuthology.orchestra.run.magna078.stderr:Read bd6edc0b/rb.0.1010.238e1f29.000000000106/head//0
2015-08-12T20:29:14.321 INFO:teuthology.orchestra.run.magna078.stderr:Read 55af3d0b/rb.0.1010.238e1f29.000000000992/head//0
2015-08-12T20:29:14.325 INFO:teuthology.orchestra.run.magna078.stderr:Read f2a86d0b/rb.0.1010.238e1f29.000000000351/head//0
2015-08-12T20:29:14.328 INFO:teuthology.orchestra.run.magna078.stderr:Read 24a34e0b/rb.0.1010.238e1f29.000000000983/head//0
2015-08-12T20:29:14.332 INFO:teuthology.orchestra.run.magna078.stderr:Read 856e811b/rb.0.1010.238e1f29.000000000d3f/head//0
2015-08-12T20:29:14.336 INFO:teuthology.orchestra.run.magna078.stderr:Read 62bf021b/rb.0.1010.238e1f29.000000000196/head//0
2015-08-12T20:29:14.376 INFO:teuthology.orchestra.run.magna078.stderr:Read b8f521b/rb.0.1010.238e1f29.0000000000f6/head//0
2015-08-12T20:29:14.377 INFO:teuthology.orchestra.run.magna078.stderr:Read f950a31b/rb.0.1010.238e1f29.000000000dac/head//0
2015-08-12T20:29:14.409 INFO:teuthology.orchestra.run.magna078.stderr:Read 503c761b/rb.0.1010.238e1f29.0000000008f7/head//0
2015-08-12T20:29:14.413 INFO:teuthology.orchestra.run.magna078.stderr:Read 5357181b/rb.0.1010.238e1f29.000000000068/head//0
2015-08-12T20:29:14.417 INFO:teuthology.orchestra.run.magna078.stderr:Read 8f7dc91b/rb.0.1010.238e1f29.000000000480/head//0
2015-08-12T20:29:14.454 INFO:teuthology.orchestra.run.magna078.stderr:Read 2f110b1b/rb.0.1010.238e1f29.0000000008e1/head//0
2015-08-12T20:29:14.498 INFO:teuthology.orchestra.run.magna078.stderr:Read e2b18c1b/rb.0.1010.238e1f29.000000000220/head//0
2015-08-12T20:29:14.542 INFO:teuthology.orchestra.run.magna078.stderr:Read 65d6ec1b/rb.0.1010.238e1f29.000000000f3b/head//0
2015-08-12T20:29:14.544 INFO:teuthology.orchestra.run.magna078.stderr:Read 50f10d1b/rb.0.1010.238e1f29.0000000009ab/head//0
2015-08-12T20:29:14.546 INFO:teuthology.orchestra.run.magna078.stderr:Read 49c42e1b/rb.0.1010.238e1f29.000000000dca/head//0
2015-08-12T20:29:14.550 INFO:teuthology.orchestra.run.magna078.stderr:Read 9939df1b/rb.0.1010.238e1f29.000000000057/head//0
2015-08-12T20:29:14.599 INFO:teuthology.orchestra.run.magna078.stderr:Read fa2f712b/rb.0.1010.238e1f29.00000000064c/head//0
2015-08-12T20:29:14.640 INFO:teuthology.orchestra.run.magna078.stderr:Read 89e0f12b/rb.0.1010.238e1f29.0000000003b7/head//0
2015-08-12T20:29:14.684 INFO:teuthology.orchestra.run.magna078.stderr:Read 8921b22b/rb.0.1010.238e1f29.00000000049c/head//0
2015-08-12T20:29:14.720 INFO:teuthology.orchestra.run.magna078.stderr:Read d61b232b/rb.0.1010.238e1f29.000000000a3c/head//0
2015-08-12T20:29:14.724 INFO:teuthology.orchestra.run.magna078.stderr:Read b327432b/rb.0.1010.238e1f29.000000000ce2/head//0
2015-08-12T20:29:14.771 INFO:teuthology.orchestra.run.magna078.stderr:Read 2fe8632b/rb.0.1010.238e1f29.000000000a0b/head//0
2015-08-12T20:29:14.814 INFO:teuthology.orchestra.run.magna078.stderr:Read 325eb32b/rb.0.1010.238e1f29.0000000003aa/head//0
2015-08-12T20:29:14.863 INFO:teuthology.orchestra.run.magna078.stderr:Read 8eace32b/rb.0.1010.238e1f29.000000000c75/head//0
2015-08-12T20:29:14.945 INFO:teuthology.orchestra.run.magna078.stderr:Read 4448e62b/rb.0.1010.238e1f29.00000000003d/head//0
2015-08-12T20:29:14.992 INFO:teuthology.orchestra.run.magna078.stderr:Read 24faf62b/rb.0.1010.238e1f29.00000000013d/head//0
2015-08-12T20:29:15.035 INFO:teuthology.orchestra.run.magna078.stderr:Read 59ca972b/rb.0.1010.238e1f29.000000000467/head//0
2015-08-12T20:29:15.078 INFO:teuthology.orchestra.run.magna078.stderr:Read 2889082b/rb.0.1010.238e1f29.000000000cf0/head//0
2015-08-12T20:29:15.082 INFO:teuthology.orchestra.run.magna078.stderr:Read 7cd3192b/rb.0.1010.238e1f29.0000000003e5/head//0
2015-08-12T20:29:15.130 INFO:teuthology.orchestra.run.magna078.stderr:Read 1fe6cb2b/rb.0.1010.238e1f29.000000000d26/head//0
2015-08-12T20:29:15.133 INFO:teuthology.orchestra.run.magna078.stderr:Read 52f71c2b/rb.0.1010.238e1f29.000000000663/head//0
2015-08-12T20:29:15.179 INFO:teuthology.orchestra.run.magna078.stderr:Read ac216e2b/rb.0.1010.238e1f29.000000000db3/head//0
2015-08-12T20:29:15.222 INFO:teuthology.orchestra.run.magna078.stderr:Read 9c3ef2b/rb.0.1010.238e1f29.0000000008d0/head//0
2015-08-12T20:29:15.226 INFO:teuthology.orchestra.run.magna078.stderr:Read 1174123b/rb.0.1010.238e1f29.00000000088c/head//0
2015-08-12T20:29:15.273 INFO:teuthology.orchestra.run.magna078.stderr:Read b6dd423b/rb.0.1010.238e1f29.000000000792/head//0
2015-08-12T20:29:15.277 INFO:teuthology.orchestra.run.magna078.stderr:Read b76a53b/rb.0.1010.238e1f29.00000000047e/head//0
2015-08-12T20:29:15.281 INFO:teuthology.orchestra.run.magna078.stderr:Read 3455d73b/rb.0.1010.238e1f29.0000000001ef/head//0
2015-08-12T20:29:15.331 INFO:teuthology.orchestra.run.magna078.stderr:Read eb89083b/rb.0.1010.238e1f29.0000000008eb/head//0
2015-08-12T20:29:15.372 INFO:teuthology.orchestra.run.magna078.stderr:Read 540ea83b/rb.0.1010.238e1f29.00000000076d/head//0
2015-08-12T20:29:15.447 INFO:teuthology.orchestra.run.magna078.stderr:Read 36c2ea3b/rb.0.1010.238e1f29.000000000d03/head//0
2015-08-12T20:29:15.451 INFO:teuthology.orchestra.run.magna078.stderr:Read eec6d3b/rb.0.1010.238e1f29.00000000081d/head//0
2015-08-12T20:29:15.455 INFO:teuthology.orchestra.run.magna078.stderr:Read 78a3cf3b/rb.0.1010.238e1f29.0000000009bf/head//0
2015-08-12T20:29:15.460 INFO:teuthology.orchestra.run.magna078.stderr:Read c612404b/rb.0.1010