Project

General

Profile

Bug #12691

ffsb osd thrash test - osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")

Added by Vasu Kulkarni over 8 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
libceph
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Seen duing rbd:thrah Can reprodouce 1/3 runs

Linux magna079 3.10.0-229.9.1.el7.x86_64 #1 SMP Sat Jul 4 17:21:43 EDT 2015 x86_64 x86_64 x86_64 GNU/Linu
2015-08-12T20:37:51.883 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.884 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.885 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.886 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.887 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.887 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.888 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.889 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.890 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.890 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.891 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.892 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.893 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.893 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:37:51.880548 7f34899e1700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f34899e1700 time 2015-08-12 20:37:51.878724
2015-08-12T20:37:51.894 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.895 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.896 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.896 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.897 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.898 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.898 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.899 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.900 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.901 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.901 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.902 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.903 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.904 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.905 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.957 INFO:tasks.ceph.osd.4.magna078.stderr:  -209> 2015-08-12 20:37:51.878713 7f34899e1700 -1 osd.4 pg_epoch: 197 pg[0.e( v 194'11837 (106'8625,194'11837] local-les=181 n=198 ec=1 les/c 181/165 180/180/173) [3,5]/[4,5] r=0 lpr=180 pi=164-179/4 rops=5 bft=3 crt=188'11821 lcod 194'11836 mlcod 194'11836 active+remapped+backfilling] bad op order, already applied 199013 > this 198728
2015-08-12T20:37:51.958 INFO:tasks.ceph.osd.4.magna078.stderr:   -21> 2015-08-12 20:37:51.880548 7f34899e1700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f34899e1700 time 2015-08-12 20:37:51.878724
2015-08-12T20:37:51.959 INFO:tasks.ceph.osd.4.magna078.stderr:osd/ReplicatedPG.cc: 2348: FAILED assert(0 == "out of order op")
2015-08-12T20:37:51.960 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.961 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.962 INFO:tasks.ceph.osd.4.magna078.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf6715]
2015-08-12T20:37:51.962 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.963 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.964 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.965 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.966 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.967 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.968 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.968 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.969 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.970 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:51.971 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.973 INFO:tasks.ceph.osd.4.magna078.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-08-12T20:37:51.973 INFO:tasks.ceph.osd.4.magna078.stderr:*** Caught signal (Aborted) **
2015-08-12T20:37:51.974 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:51.975 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.975 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:51.976 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:51.977 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:51.977 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:51.978 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:51.979 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:51.979 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:51.980 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:51.981 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:51.981 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.982 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.983 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.983 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.984 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.985 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.985 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.986 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.986 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:51.987 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:37:51.961266 7f34899e1700 -1 *** Caught signal (Aborted) **
2015-08-12T20:37:51.987 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:51.988 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:51.989 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:51.989 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:51.990 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:51.990 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:51.991 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:51.992 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:51.992 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:51.993 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:51.993 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:51.994 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:51.994 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:51.995 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:51.996 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:51.996 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:51.997 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:51.997 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:51.998 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:51.998 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:51.999 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:52.000 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:52.000 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.001 INFO:tasks.ceph.osd.4.magna078.stderr:   -41> 2015-08-12 20:37:51.961266 7f34899e1700 -1 *** Caught signal (Aborted) **
2015-08-12T20:37:52.002 INFO:tasks.ceph.osd.4.magna078.stderr: in thread 7f34899e1700
2015-08-12T20:37:52.002 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.003 INFO:tasks.ceph.osd.4.magna078.stderr: ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
2015-08-12T20:37:52.044 INFO:tasks.ceph.osd.4.magna078.stderr: 1: ceph-osd() [0x9f63f2]
2015-08-12T20:37:52.045 INFO:tasks.ceph.osd.4.magna078.stderr: 2: (()+0xf130) [0x7f34a4ee3130]
2015-08-12T20:37:52.085 INFO:tasks.ceph.osd.4.magna078.stderr: 3: (gsignal()+0x37) [0x7f34a38fd5d7]
2015-08-12T20:37:52.086 INFO:tasks.ceph.osd.4.magna078.stderr: 4: (abort()+0x148) [0x7f34a38fecc8]
2015-08-12T20:37:52.126 INFO:tasks.ceph.osd.4.magna078.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f34a42019b5]
2015-08-12T20:37:52.127 INFO:tasks.ceph.osd.4.magna078.stderr: 6: (()+0x5e926) [0x7f34a41ff926]
2015-08-12T20:37:52.167 INFO:tasks.ceph.osd.4.magna078.stderr: 7: (()+0x5e953) [0x7f34a41ff953]
2015-08-12T20:37:52.168 INFO:tasks.ceph.osd.4.magna078.stderr: 8: (()+0x5eb73) [0x7f34a41ffb73]
2015-08-12T20:37:52.208 INFO:tasks.ceph.osd.4.magna078.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xaf690a]
2015-08-12T20:37:52.209 INFO:tasks.ceph.osd.4.magna078.stderr: 10: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1914) [0x84dd24]
2015-08-12T20:37:52.249 INFO:tasks.ceph.osd.4.magna078.stderr: 11: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x45d7) [0x852677]
2015-08-12T20:37:52.250 INFO:tasks.ceph.osd.4.magna078.stderr: 12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x7ee58a]
2015-08-12T20:37:52.290 INFO:tasks.ceph.osd.4.magna078.stderr: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x652185]
2015-08-12T20:37:52.291 INFO:tasks.ceph.osd.4.magna078.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x6526a9]
2015-08-12T20:37:52.331 INFO:tasks.ceph.osd.4.magna078.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x882) [0xae5ec2]
2015-08-12T20:37:52.332 INFO:tasks.ceph.osd.4.magna078.stderr: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xae7fe0]
2015-08-12T20:37:52.333 INFO:tasks.ceph.osd.4.magna078.stderr: 17: (()+0x7df5) [0x7f34a4edbdf5]
2015-08-12T20:37:52.334 INFO:tasks.ceph.osd.4.magna078.stderr: 18: (clone()+0x6d) [0x7f34a39be1ad]
2015-08-12T20:37:52.335 INFO:tasks.ceph.osd.4.magna078.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12T20:37:52.335 INFO:tasks.ceph.osd.4.magna078.stderr:
2015-08-12T20:37:52.593 INFO:tasks.ceph.osd.4.magna078.stderr:daemon-helper: command crashed with signal 6
2015-08-12T20:37:52.939 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.4 to 1 (8655362)
2015-08-12T20:37:53.296 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1'
2015-08-12T20:37:55.333 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.5 to 1 (8655362)
2015-08-12T20:37:55.342 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1'
2015-08-12T20:37:56.806 INFO:teuthology.orchestra.run.magna087.stderr:reweighted osd.3 to 1 (8655362)
2015-08-12T20:37:56.814 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete
2015-08-12T20:37:56.815 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2015-08-12T20:37:57.090 INFO:teuthology.orchestra.run.magna087.stderr:dumped all in format json
2015-08-12T20:37:57.098 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2015-08-12T20:37:57.434 INFO:teuthology.orchestra.run.magna087.stderr:dumped all in format json
2015-08-12T20:37:57.441 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty'

Full logs

2015-08-12T20:28:44.187 INFO:teuthology.orchestra.run.magna078.stdout: marking collection for removal
2015-08-12T20:28:44.188 INFO:teuthology.orchestra.run.magna078.stdout:setting '_remove' omap key
2015-08-12T20:28:44.195 INFO:teuthology.orchestra.run.magna078.stdout:finish_remove_pgs removing 0.2_head pgid is 0.2
2015-08-12T20:28:44.196 INFO:teuthology.orchestra.run.magna078.stdout:remove_coll 0.2_head
2015-08-12T20:28:44.201 INFO:teuthology.orchestra.run.magna078.stdout:remove 2//head//0
2015-08-12T20:28:44.202 INFO:teuthology.orchestra.run.magna078.stdout:remove a05b9102/rb.0.1010.238e1f29.000000000848/head//0
2015-08-12T20:28:44.203 INFO:teuthology.orchestra.run.magna078.stdout:remove b5fe2302/rb.0.1010.238e1f29.0000000008bc/head//0
2015-08-12T20:28:44.204 INFO:teuthology.orchestra.run.magna078.stdout:remove 8131a302/rb.0.1010.238e1f29.0000000005d3/head//0
2015-08-12T20:28:44.205 INFO:teuthology.orchestra.run.magna078.stdout:remove b325a402/rb.0.1010.238e1f29.0000000003f8/head//0
2015-08-12T20:28:44.206 INFO:teuthology.orchestra.run.magna078.stdout:remove 97924602/rb.0.1010.238e1f29.00000000005e/head//0
2015-08-12T20:28:44.206 INFO:teuthology.orchestra.run.magna078.stdout:remove 7bcba602/rb.0.1010.238e1f29.00000000051e/head//0
2015-08-12T20:28:44.207 INFO:teuthology.orchestra.run.magna078.stdout:remove b2fda602/rb.0.1010.238e1f29.0000000006c6/head//0
2015-08-12T20:28:44.208 INFO:teuthology.orchestra.run.magna078.stdout:remove c053a702/rb.0.1010.238e1f29.000000000543/head//0
2015-08-12T20:28:44.209 INFO:teuthology.orchestra.run.magna078.stdout:remove f58c3802/rb.0.1010.238e1f29.000000000acd/head//0
2015-08-12T20:28:44.209 INFO:teuthology.orchestra.run.magna078.stdout:remove 96472b02/rb.0.1010.238e1f29.000000000171/head//0
2015-08-12T20:28:44.210 INFO:teuthology.orchestra.run.magna078.stdout:remove c120fc02/rb.0.1010.238e1f29.0000000005a5/head//0
2015-08-12T20:28:44.210 INFO:teuthology.orchestra.run.magna078.stdout:remove 2284ae02/rb.0.1010.238e1f29.000000000691/head//0
2015-08-12T20:28:44.211 INFO:teuthology.orchestra.run.magna078.stdout:remove db7a1f02/rb.0.1010.238e1f29.0000000005fd/head//0
2015-08-12T20:28:44.212 INFO:teuthology.orchestra.run.magna078.stdout:remove 5cba1f02/rb.0.1010.238e1f29.000000000a4e/head//0
2015-08-12T20:28:44.212 INFO:teuthology.orchestra.run.magna078.stdout:remove 5b15bf02/rb.0.1010.238e1f29.000000000087/head//0
2015-08-12T20:28:44.213 INFO:teuthology.orchestra.run.magna078.stdout:remove e8874222/rb.0.1010.238e1f29.000000000c27/head//0
2015-08-12T20:28:44.213 INFO:teuthology.orchestra.run.magna078.stdout:remove f5e15222/rb.0.1010.238e1f29.000000000786/head//0
2015-08-12T20:28:44.214 INFO:teuthology.orchestra.run.magna078.stdout:remove 6b6e1722/rb.0.1010.238e1f29.000000000cd9/head//0
2015-08-12T20:28:44.214 INFO:teuthology.orchestra.run.magna078.stdout:remove a6ea5722/rb.0.1010.238e1f29.00000000076e/head//0
2015-08-12T20:28:44.215 INFO:teuthology.orchestra.run.magna078.stdout:remove fa77a922/rb.0.1010.238e1f29.000000000141/head//0
2015-08-12T20:28:44.215 INFO:teuthology.orchestra.run.magna078.stdout:remove 4f2bb922/rb.0.1010.238e1f29.000000000262/head//0
2015-08-12T20:28:44.216 INFO:teuthology.orchestra.run.magna078.stdout:remove 36435b22/rb.0.1010.238e1f29.000000000d0c/head//0
2015-08-12T20:28:44.217 INFO:teuthology.orchestra.run.magna078.stdout:remove f071bb22/rb.0.1010.238e1f29.0000000001a4/head//0
2015-08-12T20:28:44.217 INFO:teuthology.orchestra.run.magna078.stdout:remove bba6142/rb.0.1010.238e1f29.00000000029c/head//0
2015-08-12T20:28:44.218 INFO:teuthology.orchestra.run.magna078.stdout:remove efd87142/rb.0.1010.238e1f29.000000000605/head//0
2015-08-12T20:28:44.219 INFO:teuthology.orchestra.run.magna078.stdout:remove 295db142/rb.0.1010.238e1f29.000000000105/head//0
2015-08-12T20:28:44.219 INFO:teuthology.orchestra.run.magna078.stdout:remove bbc48242/rb.0.1010.238e1f29.000000000bb7/head//0
2015-08-12T20:28:44.220 INFO:teuthology.orchestra.run.magna078.stdout:remove bafbb242/rb.0.1010.238e1f29.00000000094c/head//0
2015-08-12T20:28:44.220 INFO:teuthology.orchestra.run.magna078.stdout:remove 9e98d242/rb.0.1010.238e1f29.000000000724/head//0
2015-08-12T20:28:44.221 INFO:teuthology.orchestra.run.magna078.stdout:remove 40f97342/rb.0.1010.238e1f29.000000000806/head//0
2015-08-12T20:28:44.237 INFO:teuthology.orchestra.run.magna078.stdout:remove 7d46e342/rb.0.1010.238e1f29.0000000001a6/head//0
2015-08-12T20:28:44.239 INFO:teuthology.orchestra.run.magna078.stdout:remove aaa5a642/rb.0.1010.238e1f29.000000000dab/head//0
2015-08-12T20:28:44.240 INFO:teuthology.orchestra.run.magna078.stdout:remove 6e1f642/rb.0.1010.238e1f29.00000000020e/head//0
2015-08-12T20:28:44.241 INFO:teuthology.orchestra.run.magna078.stdout:remove 9b042742/rb.0.1010.238e1f29.0000000004b2/head//0
2015-08-12T20:28:44.241 INFO:teuthology.orchestra.run.magna078.stdout:remove 9584e842/rb.0.1010.238e1f29.000000000c17/head//0
2015-08-12T20:28:44.242 INFO:teuthology.orchestra.run.magna078.stdout:remove fe17c942/rb.0.1010.238e1f29.0000000002d2/head//0
2015-08-12T20:28:44.243 INFO:teuthology.orchestra.run.magna078.stdout:remove 5fdd4b42/rb.0.1010.238e1f29.0000000008ff/head//0
2015-08-12T20:28:44.244 INFO:teuthology.orchestra.run.magna078.stdout:remove 7f939b42/rb.0.1010.238e1f29.00000000004e/head//0
2015-08-12T20:28:44.245 INFO:teuthology.orchestra.run.magna078.stdout:remove 31166c42/rb.0.1010.238e1f29.000000000c69/head//0
2015-08-12T20:28:44.246 INFO:teuthology.orchestra.run.magna078.stdout:remove 7988cf42/rb.0.1010.238e1f29.0000000008fe/head//0
2015-08-12T20:28:44.246 INFO:teuthology.orchestra.run.magna078.stdout:remove df1d8162/rb.0.1010.238e1f29.00000000019b/head//0
2015-08-12T20:28:44.247 INFO:teuthology.orchestra.run.magna078.stdout:remove b6a2d162/rb.0.1010.238e1f29.000000000a2f/head//0
2015-08-12T20:28:44.248 INFO:teuthology.orchestra.run.magna078.stdout:remove 9ad44262/rb.0.1010.238e1f29.0000000005ed/head//0
2015-08-12T20:28:44.249 INFO:teuthology.orchestra.run.magna078.stdout:remove 9a55a262/rb.0.1010.238e1f29.00000000002c/head//0
2015-08-12T20:28:44.249 INFO:teuthology.orchestra.run.magna078.stdout:remove 5e741462/rb.0.1010.238e1f29.000000000b17/head//0
2015-08-12T20:28:44.250 INFO:teuthology.orchestra.run.magna078.stdout:remove fa1c6462/rb.0.1010.238e1f29.000000000197/head//0
2015-08-12T20:28:44.251 INFO:teuthology.orchestra.run.magna078.stdout:remove c5523562/rb.0.1010.238e1f29.000000000b77/head//0
2015-08-12T20:28:44.252 INFO:teuthology.orchestra.run.magna078.stdout:remove aa8be562/rb.0.1010.238e1f29.00000000039c/head//0
2015-08-12T20:28:44.252 INFO:teuthology.orchestra.run.magna078.stdout:remove 91659662/rb.0.1010.238e1f29.000000000b0c/head//0
2015-08-12T20:28:44.253 INFO:teuthology.orchestra.run.magna078.stdout:remove 1294d762/rb.0.1010.238e1f29.000000000505/head//0
2015-08-12T20:28:44.254 INFO:teuthology.orchestra.run.magna078.stdout:remove 1ac4bb62/rb.0.1010.238e1f29.0000000005d9/head//0
2015-08-12T20:28:44.255 INFO:teuthology.orchestra.run.magna078.stdout:remove 4fc2cb62/rb.0.1010.238e1f29.000000000563/head//0
2015-08-12T20:28:44.256 INFO:teuthology.orchestra.run.magna078.stdout:remove ab545d62/rb.0.1010.238e1f29.0000000001f9/head//0
2015-08-12T20:28:44.257 INFO:teuthology.orchestra.run.magna078.stdout:remove 6570ed62/rb.0.1010.238e1f29.000000000717/head//0
2015-08-12T20:28:44.257 INFO:teuthology.orchestra.run.magna078.stdout:remove de936e62/rb.0.1010.238e1f29.000000000668/head//0
2015-08-12T20:28:44.258 INFO:teuthology.orchestra.run.magna078.stdout:remove 46dbe62/rb.0.1010.238e1f29.000000000733/head//0
2015-08-12T20:28:44.259 INFO:teuthology.orchestra.run.magna078.stdout:remove 63954182/rb.0.1010.238e1f29.000000000607/head//0
2015-08-12T20:28:44.260 INFO:teuthology.orchestra.run.magna078.stdout:remove 61d8b282/rb.0.1010.238e1f29.000000000b5c/head//0
2015-08-12T20:28:44.261 INFO:teuthology.orchestra.run.magna078.stdout:remove 10fb7482/rb.0.1010.238e1f29.000000000f7d/head//0
2015-08-12T20:28:44.261 INFO:teuthology.orchestra.run.magna078.stdout:remove f497e482/rb.0.1010.238e1f29.000000000a15/head//0
2015-08-12T20:28:44.264 INFO:teuthology.orchestra.run.magna078.stdout:remove 9f08f482/rb.0.1010.238e1f29.000000000ba6/head//0
2015-08-12T20:28:44.265 INFO:teuthology.orchestra.run.magna078.stdout:remove af961582/rb.0.1010.238e1f29.000000000291/head//0
2015-08-12T20:28:44.266 INFO:teuthology.orchestra.run.magna078.stdout:remove fb3c3782/rb.0.1010.238e1f29.000000000239/head//0
2015-08-12T20:28:44.267 INFO:teuthology.orchestra.run.magna078.stdout:remove 98a95882/rb.0.1010.238e1f29.000000000b08/head//0
2015-08-12T20:28:44.268 INFO:teuthology.orchestra.run.magna078.stdout:remove 1e0a5982/rb.0.1010.238e1f29.00000000033e/head//0
2015-08-12T20:28:44.268 INFO:teuthology.orchestra.run.magna078.stdout:remove 111bca82/rb.0.1010.238e1f29.0000000005ef/head//0
2015-08-12T20:28:44.269 INFO:teuthology.orchestra.run.magna078.stdout:remove 9b681d82/rb.0.1010.238e1f29.000000000449/head//0
2015-08-12T20:28:44.270 INFO:teuthology.orchestra.run.magna078.stdout:remove 568ebd82/rb.0.1010.238e1f29.000000000567/head//0
2015-08-12T20:28:44.271 INFO:teuthology.orchestra.run.magna078.stdout:remove bf224e82/rb.0.1010.238e1f29.000000000a8d/head//0
2015-08-12T20:28:44.271 INFO:teuthology.orchestra.run.magna078.stdout:remove a92b3f82/rb.0.1010.238e1f29.000000000b5f/head//0
2015-08-12T20:28:44.274 INFO:teuthology.orchestra.run.magna078.stdout:remove f0ef9f82/rb.0.1010.238e1f29.0000000000cb/head//0
2015-08-12T20:28:44.275 INFO:teuthology.orchestra.run.magna078.stdout:remove 3f17e0a2/rb.0.1010.238e1f29.00000000080a/head//0
2015-08-12T20:28:44.276 INFO:teuthology.orchestra.run.magna078.stdout:remove f05791a2/rb.0.1010.238e1f29.0000000001d2/head//0
2015-08-12T20:28:44.276 INFO:teuthology.orchestra.run.magna078.stdout:remove 3364b1a2/rb.0.1010.238e1f29.0000000008fc/head//0
2015-08-12T20:28:44.277 INFO:teuthology.orchestra.run.magna078.stdout:remove 984732a2/rb.0.1010.238e1f29.000000000dc8/head//0
2015-08-12T20:28:44.278 INFO:teuthology.orchestra.run.magna078.stdout:remove 3aa4e2a2/rb.0.1010.238e1f29.000000000aa0/head//0
2015-08-12T20:28:44.279 INFO:teuthology.orchestra.run.magna078.stdout:remove 7d1af2a2/rb.0.1010.238e1f29.00000000017f/head//0
2015-08-12T20:28:44.279 INFO:teuthology.orchestra.run.magna078.stdout:remove 856fd4a2/rb.0.1010.238e1f29.000000000b1e/head//0
2015-08-12T20:28:44.280 INFO:teuthology.orchestra.run.magna078.stdout:remove af4b45a2/rb.0.1010.238e1f29.0000000008a8/head//0
2015-08-12T20:28:44.281 INFO:teuthology.orchestra.run.magna078.stdout:remove cd9be5a2/rb.0.1010.238e1f29.0000000005dc/head//0
2015-08-12T20:28:44.282 INFO:teuthology.orchestra.run.magna078.stdout:remove 76d406a2/rb.0.1010.238e1f29.000000000a45/head//0
2015-08-12T20:28:44.282 INFO:teuthology.orchestra.run.magna078.stdout:remove a9a6a8a2/rb.0.1010.238e1f29.00000000038c/head//0
2015-08-12T20:28:44.283 INFO:teuthology.orchestra.run.magna078.stdout:remove d6a819a2/rb.0.1010.238e1f29.000000000d90/head//0
2015-08-12T20:28:44.284 INFO:teuthology.orchestra.run.magna078.stdout:remove 7cf3bea2/rb.0.1010.238e1f29.00000000066b/head//0
2015-08-12T20:28:44.284 INFO:teuthology.orchestra.run.magna078.stdout:remove a4478fa2/rb.0.1010.238e1f29.0000000000a1/head//0
2015-08-12T20:28:44.285 INFO:teuthology.orchestra.run.magna078.stdout:remove 6cc0c0c2/rb.0.1010.238e1f29.0000000004b3/head//0
2015-08-12T20:28:44.286 INFO:teuthology.orchestra.run.magna078.stdout:remove e78ad1c2/rb.0.1010.238e1f29.00000000085e/head//0
2015-08-12T20:28:44.287 INFO:teuthology.orchestra.run.magna078.stdout:remove fc3b43c2/rb.0.1010.238e1f29.000000000cc5/head//0
2015-08-12T20:28:44.288 INFO:teuthology.orchestra.run.magna078.stdout:remove 43d7f5c2/rb.0.1010.238e1f29.00000000029f/head//0
2015-08-12T20:28:44.289 INFO:teuthology.orchestra.run.magna078.stdout:remove 98b8b6c2/rb.0.1010.238e1f29.000000000007/head//0
2015-08-12T20:28:44.294 INFO:teuthology.orchestra.run.magna078.stdout:remove 839427c2/rb.0.1010.238e1f29.000000000c2e/head//0
2015-08-12T20:28:44.294 INFO:teuthology.orchestra.run.magna078.stdout:remove 7cfbe7c2/rb.0.1010.238e1f29.000000000777/head//0
2015-08-12T20:28:44.295 INFO:teuthology.orchestra.run.magna078.stdout:remove 81f059c2/rb.0.1010.238e1f29.000000000486/head//0
2015-08-12T20:28:44.296 INFO:teuthology.orchestra.run.magna078.stdout:remove 62babc2/rb.0.1010.238e1f29.000000000b14/head//0
2015-08-12T20:28:44.297 INFO:teuthology.orchestra.run.magna078.stdout:remove 69ad1cc2/rb.0.1010.238e1f29.000000000497/head//0
2015-08-12T20:28:44.298 INFO:teuthology.orchestra.run.magna078.stdout:remove 858b0dc2/rb.0.1010.238e1f29.00000000047a/head//0
2015-08-12T20:28:44.298 INFO:teuthology.orchestra.run.magna078.stdout:remove 81028dc2/rb.0.1010.238e1f29.0000000008d6/head//0
2015-08-12T20:28:44.299 INFO:teuthology.orchestra.run.magna078.stdout:remove beaa8dc2/rb.0.1010.238e1f29.00000000031e/head//0
2015-08-12T20:28:44.300 INFO:teuthology.orchestra.run.magna078.stdout:remove eba2fc2/rb.0.1010.238e1f29.0000000008d7/head//0
2015-08-12T20:28:44.300 INFO:teuthology.orchestra.run.magna078.stdout:remove 7a3391e2/rb.0.1010.238e1f29.000000000742/head//0
2015-08-12T20:28:44.301 INFO:teuthology.orchestra.run.magna078.stdout:remove 623ec1e2/rb.0.1010.238e1f29.000000000460/head//0
2015-08-12T20:28:44.304 INFO:teuthology.orchestra.run.magna078.stdout:remove 2d5e43e2/rb.0.1010.238e1f29.0000000003dd/head//0
2015-08-12T20:28:44.305 INFO:teuthology.orchestra.run.magna078.stdout:remove 60d6d3e2/rb.0.1010.238e1f29.0000000009b7/head//0
2015-08-12T20:28:44.305 INFO:teuthology.orchestra.run.magna078.stdout:remove 895c04e2/rb.0.1010.238e1f29.000000000bee/head//0
2015-08-12T20:28:44.306 INFO:teuthology.orchestra.run.magna078.stdout:remove 8e9844e2/rb.0.1010.238e1f29.0000000007b1/head//0
2015-08-12T20:28:44.307 INFO:teuthology.orchestra.run.magna078.stdout:remove f290e6e2/rb.0.1010.238e1f29.00000000012e/head//0
2015-08-12T20:28:44.308 INFO:teuthology.orchestra.run.magna078.stdout:remove 80127e2/rb.0.1010.238e1f29.0000000005ae/head//0
2015-08-12T20:28:44.308 INFO:teuthology.orchestra.run.magna078.stdout:remove c92e67e2/rb.0.1010.238e1f29.0000000003f5/head//0
2015-08-12T20:28:44.309 INFO:teuthology.orchestra.run.magna078.stdout:remove e0e6e7e2/rb.0.1010.238e1f29.00000000009c/head//0
2015-08-12T20:28:44.310 INFO:teuthology.orchestra.run.magna078.stdout:remove e31638e2/rb.0.1010.238e1f29.000000000d9e/head//0
2015-08-12T20:28:44.310 INFO:teuthology.orchestra.run.magna078.stdout:remove 2e390be2/rb.0.1010.238e1f29.000000000b36/head//0
2015-08-12T20:28:44.311 INFO:teuthology.orchestra.run.magna078.stdout:remove 74d93de2/rb.0.1010.238e1f29.000000000698/head//0
2015-08-12T20:28:44.312 INFO:teuthology.orchestra.run.magna078.stdout:remove a4091ee2/rb.0.1010.238e1f29.000000000199/head//0
2015-08-12T20:28:44.312 INFO:teuthology.orchestra.run.magna078.stdout:remove b82e6ee2/rb.0.1010.238e1f29.0000000013bf/head//0
2015-08-12T20:28:44.313 INFO:teuthology.orchestra.run.magna078.stdout:remove 5311efe2/rb.0.1010.238e1f29.000000000596/head//0
2015-08-12T20:28:44.323 INFO:teuthology.orchestra.run.magna078.stdout:Remove successful
2015-08-12T20:28:44.411 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op import --file /home/ubuntu/cephtest/data/exp.0.2.3'
2015-08-12T20:28:44.779 INFO:teuthology.orchestra.run.magna078.stdout:Importing pgid 0.2
2015-08-12T20:28:44.784 INFO:teuthology.orchestra.run.magna078.stdout:Write a05b9102/rb.0.1010.238e1f29.000000000848/head//0
2015-08-12T20:28:44.836 INFO:teuthology.orchestra.run.magna078.stdout:Write b5fe2302/rb.0.1010.238e1f29.0000000008bc/head//0
2015-08-12T20:28:44.883 INFO:teuthology.orchestra.run.magna078.stdout:Write 8131a302/rb.0.1010.238e1f29.0000000005d3/head//0
2015-08-12T20:28:44.921 INFO:teuthology.orchestra.run.magna078.stdout:Write b325a402/rb.0.1010.238e1f29.0000000003f8/head//0
2015-08-12T20:28:44.967 INFO:teuthology.orchestra.run.magna078.stdout:Write 97924602/rb.0.1010.238e1f29.00000000005e/head//0
2015-08-12T20:28:45.013 INFO:teuthology.orchestra.run.magna078.stdout:Write 7bcba602/rb.0.1010.238e1f29.00000000051e/head//0
2015-08-12T20:28:45.059 INFO:teuthology.orchestra.run.magna078.stdout:Write b2fda602/rb.0.1010.238e1f29.0000000006c6/head//0
2015-08-12T20:28:45.098 INFO:teuthology.orchestra.run.magna078.stdout:Write c053a702/rb.0.1010.238e1f29.000000000543/head//0
2015-08-12T20:28:45.144 INFO:teuthology.orchestra.run.magna078.stdout:Write f58c3802/rb.0.1010.238e1f29.000000000acd/head//0
2015-08-12T20:28:45.182 INFO:teuthology.orchestra.run.magna078.stdout:Write 96472b02/rb.0.1010.238e1f29.000000000171/head//0
2015-08-12T20:28:45.228 INFO:teuthology.orchestra.run.magna078.stdout:Write c120fc02/rb.0.1010.238e1f29.0000000005a5/head//0
2015-08-12T20:28:45.357 INFO:teuthology.orchestra.run.magna078.stdout:Write 2284ae02/rb.0.1010.238e1f29.000000000691/head//0
2015-08-12T20:28:45.463 INFO:teuthology.orchestra.run.magna078.stdout:Write db7a1f02/rb.0.1010.238e1f29.0000000005fd/head//0
2015-08-12T20:28:45.568 INFO:teuthology.orchestra.run.magna078.stdout:Write 5cba1f02/rb.0.1010.238e1f29.000000000a4e/head//0
2015-08-12T20:28:45.809 INFO:teuthology.orchestra.run.magna078.stdout:Write 5b15bf02/rb.0.1010.238e1f29.000000000087/head//0
2015-08-12T20:28:45.891 INFO:teuthology.orchestra.run.magna078.stdout:Write e8874222/rb.0.1010.238e1f29.000000000c27/head//0
2015-08-12T20:28:45.994 INFO:teuthology.orchestra.run.magna078.stdout:Write f5e15222/rb.0.1010.238e1f29.000000000786/head//0
2015-08-12T20:28:46.110 INFO:teuthology.orchestra.run.magna078.stdout:Write 6b6e1722/rb.0.1010.238e1f29.000000000cd9/head//0
2015-08-12T20:28:46.182 INFO:teuthology.orchestra.run.magna078.stdout:Write a6ea5722/rb.0.1010.238e1f29.00000000076e/head//0
2015-08-12T20:28:46.302 INFO:teuthology.orchestra.run.magna078.stdout:Write fa77a922/rb.0.1010.238e1f29.000000000141/head//0
2015-08-12T20:28:46.391 INFO:teuthology.orchestra.run.magna078.stdout:Write 4f2bb922/rb.0.1010.238e1f29.000000000262/head//0
2015-08-12T20:28:46.497 INFO:teuthology.orchestra.run.magna078.stdout:Write 36435b22/rb.0.1010.238e1f29.000000000d0c/head//0
2015-08-12T20:28:46.601 INFO:teuthology.orchestra.run.magna078.stdout:Write f071bb22/rb.0.1010.238e1f29.0000000001a4/head//0
2015-08-12T20:28:46.664 INFO:teuthology.orchestra.run.magna078.stdout:Write bba6142/rb.0.1010.238e1f29.00000000029c/head//0
2015-08-12T20:28:46.800 INFO:teuthology.orchestra.run.magna078.stdout:Write efd87142/rb.0.1010.238e1f29.000000000605/head//0
2015-08-12T20:28:46.850 INFO:teuthology.orchestra.run.magna078.stdout:Write 295db142/rb.0.1010.238e1f29.000000000105/head//0
2015-08-12T20:28:46.888 INFO:teuthology.orchestra.run.magna078.stdout:Write bbc48242/rb.0.1010.238e1f29.000000000bb7/head//0
2015-08-12T20:28:47.091 INFO:teuthology.orchestra.run.magna078.stdout:Write bafbb242/rb.0.1010.238e1f29.00000000094c/head//0
2015-08-12T20:28:47.315 INFO:teuthology.orchestra.run.magna078.stdout:Write 9e98d242/rb.0.1010.238e1f29.000000000724/head//0
2015-08-12T20:28:47.551 INFO:teuthology.orchestra.run.magna078.stdout:Write 40f97342/rb.0.1010.238e1f29.000000000806/head//0
2015-08-12T20:28:47.691 INFO:teuthology.orchestra.run.magna078.stdout:Write 7d46e342/rb.0.1010.238e1f29.0000000001a6/head//0
2015-08-12T20:28:47.751 INFO:teuthology.orchestra.run.magna078.stdout:Write aaa5a642/rb.0.1010.238e1f29.000000000dab/head//0
2015-08-12T20:28:48.283 INFO:teuthology.orchestra.run.magna078.stdout:Write 6e1f642/rb.0.1010.238e1f29.00000000020e/head//0
2015-08-12T20:28:48.323 INFO:teuthology.orchestra.run.magna078.stdout:Write 9b042742/rb.0.1010.238e1f29.0000000004b2/head//0
2015-08-12T20:28:48.324 INFO:teuthology.orchestra.run.magna078.stdout:Write 9584e842/rb.0.1010.238e1f29.000000000c17/head//0
2015-08-12T20:28:48.324 INFO:teuthology.orchestra.run.magna078.stdout:Write fe17c942/rb.0.1010.238e1f29.0000000002d2/head//0
2015-08-12T20:28:48.325 INFO:teuthology.orchestra.run.magna078.stdout:Write 5fdd4b42/rb.0.1010.238e1f29.0000000008ff/head//0
2015-08-12T20:28:48.326 INFO:teuthology.orchestra.run.magna078.stdout:Write 7f939b42/rb.0.1010.238e1f29.00000000004e/head//0
2015-08-12T20:28:48.327 INFO:teuthology.orchestra.run.magna078.stdout:Write 31166c42/rb.0.1010.238e1f29.000000000c69/head//0
2015-08-12T20:28:48.327 INFO:teuthology.orchestra.run.magna078.stdout:Write 7988cf42/rb.0.1010.238e1f29.0000000008fe/head//0
2015-08-12T20:28:48.378 INFO:teuthology.orchestra.run.magna078.stdout:Write df1d8162/rb.0.1010.238e1f29.00000000019b/head//0
2015-08-12T20:28:48.716 INFO:teuthology.orchestra.run.magna078.stdout:Write b6a2d162/rb.0.1010.238e1f29.000000000a2f/head//0
2015-08-12T20:28:48.850 INFO:teuthology.orchestra.run.magna078.stdout:Write 9ad44262/rb.0.1010.238e1f29.0000000005ed/head//0
2015-08-12T20:28:48.925 INFO:teuthology.orchestra.run.magna078.stdout:Write 9a55a262/rb.0.1010.238e1f29.00000000002c/head//0
2015-08-12T20:28:48.971 INFO:teuthology.orchestra.run.magna078.stdout:Write 5e741462/rb.0.1010.238e1f29.000000000b17/head//0
2015-08-12T20:28:49.010 INFO:teuthology.orchestra.run.magna078.stdout:Write fa1c6462/rb.0.1010.238e1f29.000000000197/head//0
2015-08-12T20:28:49.065 INFO:teuthology.orchestra.run.magna078.stdout:Write c5523562/rb.0.1010.238e1f29.000000000b77/head//0
2015-08-12T20:28:49.111 INFO:teuthology.orchestra.run.magna078.stdout:Write aa8be562/rb.0.1010.238e1f29.00000000039c/head//0
2015-08-12T20:28:49.144 INFO:teuthology.orchestra.run.magna078.stdout:Write 91659662/rb.0.1010.238e1f29.000000000b0c/head//0
2015-08-12T20:28:49.200 INFO:teuthology.orchestra.run.magna078.stdout:Write 1294d762/rb.0.1010.238e1f29.000000000505/head//0
2015-08-12T20:28:49.247 INFO:teuthology.orchestra.run.magna078.stdout:Write 1ac4bb62/rb.0.1010.238e1f29.0000000005d9/head//0
2015-08-12T20:28:49.293 INFO:teuthology.orchestra.run.magna078.stdout:Write 4fc2cb62/rb.0.1010.238e1f29.000000000563/head//0
2015-08-12T20:28:49.331 INFO:teuthology.orchestra.run.magna078.stdout:Write ab545d62/rb.0.1010.238e1f29.0000000001f9/head//0
2015-08-12T20:28:49.687 INFO:teuthology.orchestra.run.magna078.stdout:Write 6570ed62/rb.0.1010.238e1f29.000000000717/head//0
2015-08-12T20:28:49.908 INFO:teuthology.orchestra.run.magna078.stdout:Write de936e62/rb.0.1010.238e1f29.000000000668/head//0
2015-08-12T20:28:49.987 INFO:teuthology.orchestra.run.magna078.stdout:Write 46dbe62/rb.0.1010.238e1f29.000000000733/head//0
2015-08-12T20:28:50.059 INFO:teuthology.orchestra.run.magna078.stdout:Write 63954182/rb.0.1010.238e1f29.000000000607/head//0
2015-08-12T20:28:50.149 INFO:teuthology.orchestra.run.magna078.stdout:Write 61d8b282/rb.0.1010.238e1f29.000000000b5c/head//0
2015-08-12T20:28:50.236 INFO:teuthology.orchestra.run.magna078.stdout:Write 10fb7482/rb.0.1010.238e1f29.000000000f7d/head//0
2015-08-12T20:28:50.266 INFO:teuthology.orchestra.run.magna078.stdout:Write f497e482/rb.0.1010.238e1f29.000000000a15/head//0
2015-08-12T20:28:50.379 INFO:teuthology.orchestra.run.magna078.stdout:Write 9f08f482/rb.0.1010.238e1f29.000000000ba6/head//0
2015-08-12T20:28:50.460 INFO:teuthology.orchestra.run.magna078.stdout:Write af961582/rb.0.1010.238e1f29.000000000291/head//0
2015-08-12T20:28:50.540 INFO:teuthology.orchestra.run.magna078.stdout:Write fb3c3782/rb.0.1010.238e1f29.000000000239/head//0
2015-08-12T20:28:50.587 INFO:teuthology.orchestra.run.magna078.stdout:Write 98a95882/rb.0.1010.238e1f29.000000000b08/head//0
2015-08-12T20:28:50.634 INFO:teuthology.orchestra.run.magna078.stdout:Write 1e0a5982/rb.0.1010.238e1f29.00000000033e/head//0
2015-08-12T20:28:50.680 INFO:teuthology.orchestra.run.magna078.stdout:Write 111bca82/rb.0.1010.238e1f29.0000000005ef/head//0
2015-08-12T20:28:50.718 INFO:teuthology.orchestra.run.magna078.stdout:Write 9b681d82/rb.0.1010.238e1f29.000000000449/head//0
2015-08-12T20:28:50.864 INFO:teuthology.orchestra.run.magna078.stdout:Write 568ebd82/rb.0.1010.238e1f29.000000000567/head//0
2015-08-12T20:28:51.015 INFO:teuthology.orchestra.run.magna078.stdout:Write bf224e82/rb.0.1010.238e1f29.000000000a8d/head//0
2015-08-12T20:28:51.198 INFO:teuthology.orchestra.run.magna078.stdout:Write a92b3f82/rb.0.1010.238e1f29.000000000b5f/head//0
2015-08-12T20:28:51.398 INFO:teuthology.orchestra.run.magna078.stdout:Write f0ef9f82/rb.0.1010.238e1f29.0000000000cb/head//0
2015-08-12T20:28:51.547 INFO:teuthology.orchestra.run.magna078.stdout:Write 3f17e0a2/rb.0.1010.238e1f29.00000000080a/head//0
2015-08-12T20:28:51.609 INFO:teuthology.orchestra.run.magna078.stdout:Write f05791a2/rb.0.1010.238e1f29.0000000001d2/head//0
2015-08-12T20:28:51.683 INFO:teuthology.orchestra.run.magna078.stdout:Write 3364b1a2/rb.0.1010.238e1f29.0000000008fc/head//0
2015-08-12T20:28:51.730 INFO:teuthology.orchestra.run.magna078.stdout:Write 984732a2/rb.0.1010.238e1f29.000000000dc8/head//0
2015-08-12T20:28:51.784 INFO:teuthology.orchestra.run.magna078.stdout:Write 3aa4e2a2/rb.0.1010.238e1f29.000000000aa0/head//0
2015-08-12T20:28:51.820 INFO:teuthology.orchestra.run.magna078.stdout:Write 7d1af2a2/rb.0.1010.238e1f29.00000000017f/head//0
2015-08-12T20:28:51.858 INFO:teuthology.orchestra.run.magna078.stdout:Write 856fd4a2/rb.0.1010.238e1f29.000000000b1e/head//0
2015-08-12T20:28:51.896 INFO:teuthology.orchestra.run.magna078.stdout:Write af4b45a2/rb.0.1010.238e1f29.0000000008a8/head//0
2015-08-12T20:28:51.934 INFO:teuthology.orchestra.run.magna078.stdout:Write cd9be5a2/rb.0.1010.238e1f29.0000000005dc/head//0
2015-08-12T20:28:52.209 INFO:teuthology.orchestra.run.magna078.stdout:Write 76d406a2/rb.0.1010.238e1f29.000000000a45/head//0
2015-08-12T20:28:52.492 INFO:teuthology.orchestra.run.magna078.stdout:Write a9a6a8a2/rb.0.1010.238e1f29.00000000038c/head//0
2015-08-12T20:28:52.644 INFO:teuthology.orchestra.run.magna078.stdout:Write d6a819a2/rb.0.1010.238e1f29.000000000d90/head//0
2015-08-12T20:28:52.700 INFO:teuthology.orchestra.run.magna078.stdout:Write 7cf3bea2/rb.0.1010.238e1f29.00000000066b/head//0
2015-08-12T20:28:52.766 INFO:teuthology.orchestra.run.magna078.stdout:Write a4478fa2/rb.0.1010.238e1f29.0000000000a1/head//0
2015-08-12T20:28:53.283 INFO:teuthology.orchestra.run.magna078.stdout:Write 6cc0c0c2/rb.0.1010.238e1f29.0000000004b3/head//0
2015-08-12T20:28:53.322 INFO:teuthology.orchestra.run.magna078.stdout:Write e78ad1c2/rb.0.1010.238e1f29.00000000085e/head//0
2015-08-12T20:28:53.323 INFO:teuthology.orchestra.run.magna078.stdout:Write fc3b43c2/rb.0.1010.238e1f29.000000000cc5/head//0
2015-08-12T20:28:53.364 INFO:teuthology.orchestra.run.magna078.stdout:Write 43d7f5c2/rb.0.1010.238e1f29.00000000029f/head//0
2015-08-12T20:28:53.365 INFO:teuthology.orchestra.run.magna078.stdout:Write 98b8b6c2/rb.0.1010.238e1f29.000000000007/head//0
2015-08-12T20:28:53.406 INFO:teuthology.orchestra.run.magna078.stdout:Write 839427c2/rb.0.1010.238e1f29.000000000c2e/head//0
2015-08-12T20:28:53.407 INFO:teuthology.orchestra.run.magna078.stdout:Write 7cfbe7c2/rb.0.1010.238e1f29.000000000777/head//0
2015-08-12T20:28:53.447 INFO:teuthology.orchestra.run.magna078.stdout:Write 81f059c2/rb.0.1010.238e1f29.000000000486/head//0
2015-08-12T20:28:53.448 INFO:teuthology.orchestra.run.magna078.stdout:Write 62babc2/rb.0.1010.238e1f29.000000000b14/head//0
2015-08-12T20:28:53.489 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.462915 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.462914)
2015-08-12T20:28:53.583 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.575237 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.575237)
2015-08-12T20:28:53.666 INFO:teuthology.orchestra.run.magna078.stdout:Write 69ad1cc2/rb.0.1010.238e1f29.000000000497/head//0
2015-08-12T20:28:53.766 INFO:teuthology.orchestra.run.magna078.stdout:Write 858b0dc2/rb.0.1010.238e1f29.00000000047a/head//0
2015-08-12T20:28:53.837 INFO:teuthology.orchestra.run.magna078.stdout:Write 81028dc2/rb.0.1010.238e1f29.0000000008d6/head//0
2015-08-12T20:28:53.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:53.856988 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:33.856987)
2015-08-12T20:28:53.908 INFO:teuthology.orchestra.run.magna078.stdout:Write beaa8dc2/rb.0.1010.238e1f29.00000000031e/head//0
2015-08-12T20:28:53.971 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:53.963196 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:33.963195)
2015-08-12T20:28:54.036 INFO:teuthology.orchestra.run.magna078.stdout:Write eba2fc2/rb.0.1010.238e1f29.0000000008d7/head//0
2015-08-12T20:28:54.131 INFO:teuthology.orchestra.run.magna078.stdout:Write 7a3391e2/rb.0.1010.238e1f29.000000000742/head//0
2015-08-12T20:28:54.154 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:54.145905 7f2e489a7700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:34.145903)
2015-08-12T20:28:54.212 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:54.211687 7ff3458aa700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:34.211686)
2015-08-12T20:28:54.225 INFO:teuthology.orchestra.run.magna078.stdout:Write 623ec1e2/rb.0.1010.238e1f29.000000000460/head//0
2015-08-12T20:28:54.278 INFO:teuthology.orchestra.run.magna078.stdout:Write 2d5e43e2/rb.0.1010.238e1f29.0000000003dd/head//0
2015-08-12T20:28:54.324 INFO:teuthology.orchestra.run.magna078.stdout:Write 60d6d3e2/rb.0.1010.238e1f29.0000000009b7/head//0
2015-08-12T20:28:54.362 INFO:teuthology.orchestra.run.magna078.stdout:Write 895c04e2/rb.0.1010.238e1f29.000000000bee/head//0
2015-08-12T20:28:54.399 INFO:teuthology.orchestra.run.magna078.stdout:Write 8e9844e2/rb.0.1010.238e1f29.0000000007b1/head//0
2015-08-12T20:28:54.439 INFO:teuthology.orchestra.run.magna078.stdout:Write f290e6e2/rb.0.1010.238e1f29.00000000012e/head//0
2015-08-12T20:28:54.583 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:54.575467 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:34.575466)
2015-08-12T20:28:54.602 INFO:teuthology.orchestra.run.magna078.stdout:Write 80127e2/rb.0.1010.238e1f29.0000000005ae/head//0
2015-08-12T20:28:54.744 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:54.743771 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:34.743769)
2015-08-12T20:28:54.771 INFO:teuthology.orchestra.run.magna078.stdout:Write c92e67e2/rb.0.1010.238e1f29.0000000003f5/head//0
2015-08-12T20:28:54.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:54.857136 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:34.857135)
2015-08-12T20:28:54.986 INFO:teuthology.orchestra.run.magna078.stdout:Write e0e6e7e2/rb.0.1010.238e1f29.00000000009c/head//0
2015-08-12T20:28:55.130 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:55.129858 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:35.129856)
2015-08-12T20:28:55.200 INFO:teuthology.orchestra.run.magna078.stdout:Write e31638e2/rb.0.1010.238e1f29.000000000d9e/head//0
2015-08-12T20:28:55.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.203738 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.203737)
2015-08-12T20:28:55.338 INFO:teuthology.orchestra.run.magna078.stdout:Write 2e390be2/rb.0.1010.238e1f29.000000000b36/head//0
2015-08-12T20:28:55.408 INFO:teuthology.orchestra.run.magna078.stdout:Write 74d93de2/rb.0.1010.238e1f29.000000000698/head//0
2015-08-12T20:28:55.430 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.422616 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.422615)
2015-08-12T20:28:55.454 INFO:teuthology.orchestra.run.magna078.stdout:Write a4091ee2/rb.0.1010.238e1f29.000000000199/head//0
2015-08-12T20:28:55.500 INFO:teuthology.orchestra.run.magna078.stdout:Write b82e6ee2/rb.0.1010.238e1f29.0000000013bf/head//0
2015-08-12T20:28:55.523 INFO:teuthology.orchestra.run.magna078.stdout:Write 5311efe2/rb.0.1010.238e1f29.000000000596/head//0
2015-08-12T20:28:55.532 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:55.531059 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:35.531058)
2015-08-12T20:28:55.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:55.575630 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:35.575629)
2015-08-12T20:28:55.606 INFO:teuthology.orchestra.run.magna078.stdout:Import successful
2015-08-12T20:28:55.744 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:55.743939 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:35.743937)
2015-08-12T20:28:55.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:55.857233 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:35.857232)
2015-08-12T20:28:55.931 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:55.922930 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:35.922930)
2015-08-12T20:28:56.104 INFO:teuthology.orchestra.run.magna078:Running: 'rm -f /home/ubuntu/cephtest/data/exp.0.2.3'
2015-08-12T20:28:56.131 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:56.130115 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:36.130113)
2015-08-12T20:28:56.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:56.203887 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:36.203887)
2015-08-12T20:28:56.271 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:56.263494 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:36.263494)
2015-08-12T20:28:56.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:56.575772 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:36.575771)
2015-08-12T20:28:56.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:56.744048 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:36.744046)
2015-08-12T20:28:56.865 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:56.857381 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:36.857380)
2015-08-12T20:28:57.031 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:57.023248 7f23cc6cb700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:37.023247)
2015-08-12T20:28:57.112 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:57.112016 7ff3458aa700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:37.112016)
2015-08-12T20:28:57.131 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:57.130278 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:37.130277)
2015-08-12T20:28:57.212 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:57.204051 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:37.204050)
2015-08-12T20:28:57.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:57.575918 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:37.575917)
2015-08-12T20:28:57.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:57.744158 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:37.744157)
2015-08-12T20:28:58.256 INFO:tasks.ceph.osd.0.magna087.stderr:2015-08-12 20:28:57.857483 7f2e60340700 -1 osd.0 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.644187 front 2015-08-12 20:28:33.644187 (cutoff 2015-08-12 20:28:37.857483)
2015-08-12T20:28:58.295 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:57.963853 7f32c7fb8700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:37.963852)
2015-08-12T20:28:58.296 INFO:tasks.ceph.osd.2.magna087.stderr:2015-08-12 20:28:58.204288 7f23e4064700 -1 osd.2 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.420598 front 2015-08-12 20:28:34.420598 (cutoff 2015-08-12 20:28:38.204287)
2015-08-12T20:28:58.337 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:57.831342 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:37.831341)
2015-08-12T20:28:58.338 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:58.130474 7f349cb71700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:38.130473)
2015-08-12T20:28:58.339 INFO:tasks.ceph.osd.4.magna078.stderr:2015-08-12 20:28:58.331587 7f34851d8700 -1 osd.4 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:34.528987 front 2015-08-12 20:28:34.528987 (cutoff 2015-08-12 20:28:38.331587)
2015-08-12T20:28:58.584 INFO:tasks.ceph.osd.1.magna087.stderr:2015-08-12 20:28:58.576018 7f32df951700 -1 osd.1 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:32.961327 front 2015-08-12 20:28:32.961327 (cutoff 2015-08-12 20:28:38.576018)
2015-08-12T20:28:58.745 INFO:tasks.ceph.osd.5.magna078.stderr:2015-08-12 20:28:58.744337 7ff35d243700 -1 osd.5 95 heartbeat_check: no reply from osd.3 since back 2015-08-12 20:28:33.809596 front 2015-08-12 20:28:33.809596 (cutoff 2015-08-12 20:28:38.744336)
2015-08-12T20:29:01.868 INFO:tasks.thrashosds.thrasher:in_osds:  [3, 2, 4, 1, 5, 0] out_osds:  [] dead_osds:  [3] live_osds:  [0, 2, 4, 1, 5]
2015-08-12T20:29:01.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-08-12T20:29:01.871 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [3, 2, 4, 1, 5, 0]
2015-08-12T20:29:01.872 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3'
2015-08-12T20:29:02.429 INFO:teuthology.orchestra.run.magna087.stderr:marked out osd.3.
2015-08-12T20:29:07.437 INFO:tasks.thrashosds.thrasher:in_osds:  [2, 4, 1, 5, 0] out_osds:  [3] dead_osds:  [3] live_osds:  [0, 2, 4, 1, 5]
2015-08-12T20:29:07.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-08-12T20:29:07.439 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 4, 1, 5]
2015-08-12T20:29:13.438 INFO:tasks.ceph.osd.2:Stopped
2015-08-12T20:29:13.439 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd
2015-08-12T20:29:13.440 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
2015-08-12T20:29:13.768 INFO:teuthology.orchestra.run.magna078:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 0.b --file /home/ubuntu/cephtest/data/exp.0.b.3'
2015-08-12T20:29:13.970 INFO:teuthology.orchestra.run.magna078.stderr:Exporting 0.b
2015-08-12T20:29:14.024 INFO:teuthology.orchestra.run.magna078.stderr:Read 3e47000b/rb.0.1010.238e1f29.0000000000ac/head//0
2015-08-12T20:29:14.072 INFO:teuthology.orchestra.run.magna078.stderr:Read b1bbb00b/rb.0.1010.238e1f29.000000000927/head//0
2015-08-12T20:29:14.077 INFO:teuthology.orchestra.run.magna078.stderr:Read d7f1f00b/rb.0.1010.238e1f29.000000000ac5/head//0
2015-08-12T20:29:14.081 INFO:teuthology.orchestra.run.magna078.stderr:Read b189f00b/rb.0.1010.238e1f29.0000000003bf/head//0
2015-08-12T20:29:14.085 INFO:teuthology.orchestra.run.magna078.stderr:Read b5e8510b/rb.0.1010.238e1f29.0000000003e9/head//0
2015-08-12T20:29:14.089 INFO:teuthology.orchestra.run.magna078.stderr:Read a8d2420b/rb.0.1010.238e1f29.000000000ccd/head//0
2015-08-12T20:29:14.093 INFO:teuthology.orchestra.run.magna078.stderr:Read 4cadd20b/rb.0.1010.238e1f29.00000000012a/head//0
2015-08-12T20:29:14.128 INFO:teuthology.orchestra.run.magna078.stderr:Read 15f8940b/rb.0.1010.238e1f29.000000000a67/head//0
2015-08-12T20:29:14.171 INFO:teuthology.orchestra.run.magna078.stderr:Read 32e9350b/rb.0.1010.238e1f29.00000000047b/head//0
2015-08-12T20:29:14.175 INFO:teuthology.orchestra.run.magna078.stderr:Read 47b850b/rb.0.1010.238e1f29.000000000692/head//0
2015-08-12T20:29:14.217 INFO:teuthology.orchestra.run.magna078.stderr:Read 2a57570b/rb.0.1010.238e1f29.00000000063d/head//0
2015-08-12T20:29:14.266 INFO:teuthology.orchestra.run.magna078.stderr:Read 908e80b/rb.0.1010.238e1f29.000000000325/head//0
2015-08-12T20:29:14.270 INFO:teuthology.orchestra.run.magna078.stderr:Read 4e17390b/rb.0.1010.238e1f29.000000000a41/head//0
2015-08-12T20:29:14.316 INFO:teuthology.orchestra.run.magna078.stderr:Read bd6edc0b/rb.0.1010.238e1f29.000000000106/head//0
2015-08-12T20:29:14.321 INFO:teuthology.orchestra.run.magna078.stderr:Read 55af3d0b/rb.0.1010.238e1f29.000000000992/head//0
2015-08-12T20:29:14.325 INFO:teuthology.orchestra.run.magna078.stderr:Read f2a86d0b/rb.0.1010.238e1f29.000000000351/head//0
2015-08-12T20:29:14.328 INFO:teuthology.orchestra.run.magna078.stderr:Read 24a34e0b/rb.0.1010.238e1f29.000000000983/head//0
2015-08-12T20:29:14.332 INFO:teuthology.orchestra.run.magna078.stderr:Read 856e811b/rb.0.1010.238e1f29.000000000d3f/head//0
2015-08-12T20:29:14.336 INFO:teuthology.orchestra.run.magna078.stderr:Read 62bf021b/rb.0.1010.238e1f29.000000000196/head//0
2015-08-12T20:29:14.376 INFO:teuthology.orchestra.run.magna078.stderr:Read b8f521b/rb.0.1010.238e1f29.0000000000f6/head//0
2015-08-12T20:29:14.377 INFO:teuthology.orchestra.run.magna078.stderr:Read f950a31b/rb.0.1010.238e1f29.000000000dac/head//0
2015-08-12T20:29:14.409 INFO:teuthology.orchestra.run.magna078.stderr:Read 503c761b/rb.0.1010.238e1f29.0000000008f7/head//0
2015-08-12T20:29:14.413 INFO:teuthology.orchestra.run.magna078.stderr:Read 5357181b/rb.0.1010.238e1f29.000000000068/head//0
2015-08-12T20:29:14.417 INFO:teuthology.orchestra.run.magna078.stderr:Read 8f7dc91b/rb.0.1010.238e1f29.000000000480/head//0
2015-08-12T20:29:14.454 INFO:teuthology.orchestra.run.magna078.stderr:Read 2f110b1b/rb.0.1010.238e1f29.0000000008e1/head//0
2015-08-12T20:29:14.498 INFO:teuthology.orchestra.run.magna078.stderr:Read e2b18c1b/rb.0.1010.238e1f29.000000000220/head//0
2015-08-12T20:29:14.542 INFO:teuthology.orchestra.run.magna078.stderr:Read 65d6ec1b/rb.0.1010.238e1f29.000000000f3b/head//0
2015-08-12T20:29:14.544 INFO:teuthology.orchestra.run.magna078.stderr:Read 50f10d1b/rb.0.1010.238e1f29.0000000009ab/head//0
2015-08-12T20:29:14.546 INFO:teuthology.orchestra.run.magna078.stderr:Read 49c42e1b/rb.0.1010.238e1f29.000000000dca/head//0
2015-08-12T20:29:14.550 INFO:teuthology.orchestra.run.magna078.stderr:Read 9939df1b/rb.0.1010.238e1f29.000000000057/head//0
2015-08-12T20:29:14.599 INFO:teuthology.orchestra.run.magna078.stderr:Read fa2f712b/rb.0.1010.238e1f29.00000000064c/head//0
2015-08-12T20:29:14.640 INFO:teuthology.orchestra.run.magna078.stderr:Read 89e0f12b/rb.0.1010.238e1f29.0000000003b7/head//0
2015-08-12T20:29:14.684 INFO:teuthology.orchestra.run.magna078.stderr:Read 8921b22b/rb.0.1010.238e1f29.00000000049c/head//0
2015-08-12T20:29:14.720 INFO:teuthology.orchestra.run.magna078.stderr:Read d61b232b/rb.0.1010.238e1f29.000000000a3c/head//0
2015-08-12T20:29:14.724 INFO:teuthology.orchestra.run.magna078.stderr:Read b327432b/rb.0.1010.238e1f29.000000000ce2/head//0
2015-08-12T20:29:14.771 INFO:teuthology.orchestra.run.magna078.stderr:Read 2fe8632b/rb.0.1010.238e1f29.000000000a0b/head//0
2015-08-12T20:29:14.814 INFO:teuthology.orchestra.run.magna078.stderr:Read 325eb32b/rb.0.1010.238e1f29.0000000003aa/head//0
2015-08-12T20:29:14.863 INFO:teuthology.orchestra.run.magna078.stderr:Read 8eace32b/rb.0.1010.238e1f29.000000000c75/head//0
2015-08-12T20:29:14.945 INFO:teuthology.orchestra.run.magna078.stderr:Read 4448e62b/rb.0.1010.238e1f29.00000000003d/head//0
2015-08-12T20:29:14.992 INFO:teuthology.orchestra.run.magna078.stderr:Read 24faf62b/rb.0.1010.238e1f29.00000000013d/head//0
2015-08-12T20:29:15.035 INFO:teuthology.orchestra.run.magna078.stderr:Read 59ca972b/rb.0.1010.238e1f29.000000000467/head//0
2015-08-12T20:29:15.078 INFO:teuthology.orchestra.run.magna078.stderr:Read 2889082b/rb.0.1010.238e1f29.000000000cf0/head//0
2015-08-12T20:29:15.082 INFO:teuthology.orchestra.run.magna078.stderr:Read 7cd3192b/rb.0.1010.238e1f29.0000000003e5/head//0
2015-08-12T20:29:15.130 INFO:teuthology.orchestra.run.magna078.stderr:Read 1fe6cb2b/rb.0.1010.238e1f29.000000000d26/head//0
2015-08-12T20:29:15.133 INFO:teuthology.orchestra.run.magna078.stderr:Read 52f71c2b/rb.0.1010.238e1f29.000000000663/head//0
2015-08-12T20:29:15.179 INFO:teuthology.orchestra.run.magna078.stderr:Read ac216e2b/rb.0.1010.238e1f29.000000000db3/head//0
2015-08-12T20:29:15.222 INFO:teuthology.orchestra.run.magna078.stderr:Read 9c3ef2b/rb.0.1010.238e1f29.0000000008d0/head//0
2015-08-12T20:29:15.226 INFO:teuthology.orchestra.run.magna078.stderr:Read 1174123b/rb.0.1010.238e1f29.00000000088c/head//0
2015-08-12T20:29:15.273 INFO:teuthology.orchestra.run.magna078.stderr:Read b6dd423b/rb.0.1010.238e1f29.000000000792/head//0
2015-08-12T20:29:15.277 INFO:teuthology.orchestra.run.magna078.stderr:Read b76a53b/rb.0.1010.238e1f29.00000000047e/head//0
2015-08-12T20:29:15.281 INFO:teuthology.orchestra.run.magna078.stderr:Read 3455d73b/rb.0.1010.238e1f29.0000000001ef/head//0
2015-08-12T20:29:15.331 INFO:teuthology.orchestra.run.magna078.stderr:Read eb89083b/rb.0.1010.238e1f29.0000000008eb/head//0
2015-08-12T20:29:15.372 INFO:teuthology.orchestra.run.magna078.stderr:Read 540ea83b/rb.0.1010.238e1f29.00000000076d/head//0
2015-08-12T20:29:15.447 INFO:teuthology.orchestra.run.magna078.stderr:Read 36c2ea3b/rb.0.1010.238e1f29.000000000d03/head//0
2015-08-12T20:29:15.451 INFO:teuthology.orchestra.run.magna078.stderr:Read eec6d3b/rb.0.1010.238e1f29.00000000081d/head//0
2015-08-12T20:29:15.455 INFO:teuthology.orchestra.run.magna078.stderr:Read 78a3cf3b/rb.0.1010.238e1f29.0000000009bf/head//0
2015-08-12T20:29:15.460 INFO:teuthology.orchestra.run.magna078.stderr:Read c612404b/rb.0.1010

osd_logs.tar.gz - logs containing two relevant ops on osd.1 and osd.3 (24 KB) Josh Durgin, 09/16/2015 07:28 AM


Related issues

Related to Linux kernel client - Bug #10889: krbd: sent out of order write Resolved 02/16/2015

History

#1 Updated by Josh Durgin over 8 years ago

Can you post the yaml config for this test? We haven't seen it in the usual nightlies, and it'd be good to add it there if it's that reproducible.

#2 Updated by Josh Durgin over 8 years ago

  • Project changed from rbd to Linux kernel client
  • Category set to rbd

#3 Updated by Ilya Dryomov over 8 years ago

  • Status changed from New to Need More Info

#4 Updated by Ilya Dryomov over 8 years ago

A thrash means lingering requests code is in play, which is a large part of what I'm rewriting for 4.4. Still, it would be good to track this down in the old code to make sure it's taken care of. The posted teuthology log is pretty useless, so I need a yaml file to try to reproduce.

#5 Updated by Vasu Kulkarni over 8 years ago

The logs I attached was to show the backtrace, but here is the test that I am running from ceph-qa-suites on rh 7.1 , you just need the below filter for the krbd/trash suite.

description: krbd:thrash/{conf.yaml clusters/fixed-3.yaml fs/xfs.yaml thrashers/default.yaml
workloads/rbd_workunit_suites_ffsb.yaml}

I am trying to run this again using the krbd client side debug enabled, will update when I can rereate this.

#6 Updated by Vasu Kulkarni over 8 years ago

Following is the config file, you might have to modify it based on the nodes and os-type you reserve, this config has the debug which josh wanted to enable

os_type: rhel
interactive-on-error: True
os_version: '7.1'
overrides:
  ansible:
    cleanup: true
    playbook: downstream_setup.yml
    repo: https://github.com/ceph/ceph-cm-ansible.git
    vars:
      disable_yum_repos:
      - epel
      - rhel7
      - rhel7-optional
      - rhel7-extras
      - lab-extras
      - rhel-7-fcgi-ceph
      yum_repos:
      - name: all_repo
        url: http://fpaste.org/244005/43682819/raw/
  ceph:
    conf:
      global:
        ms die on skipped message: false
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
        osd op thread timeout: 60
        osd sloppy crc: true
    fs: xfs
    log-whitelist:
    - slow request
  install:
  workunit:
    tag: v0.94.1
owner: scheduled_vasu@magna002
rhversion: 1.3.0
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - osd.3
  - osd.4
  - osd.5
- - client.0
suite: krbd:thrash
suite_branch: rh130
suite_path: /home/teuthworker/src/ceph-qa-suite_rh130
targets:
  magna078.ceph.redhat.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDagT5KsRV2+DWEGgKfDQRPsRdD7+yijlmONAkpYcpIs8ylxNxno6t0XbbvnWI7JpUVYmqrFD64Xdos1+bJo7YSMhNhH4+c4UlRCx8TE2KOvbBzsAQz1WBU8ZNP1pZbH+5ErjPRkPoFhY/DyNCk0I3qUuucI2nSIcTBYR+vfrxVgecd9dUUVjgsOTts0ATyTO67oxqFSQPP34ZWcN2VQJKtFeJZN33LvjAAqlHBD5blnxgptrcMH/0IHJwO+fxzAq3OIxTDX9GeQP9Dm2y2VQhHABGL24zkcwOwvLVSnoSDXbB1tkTDPeoRNhZOcQS5KaPvP0TplO7YT1gj0DfET0jx
  magna079.ceph.redhat.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsudTthyOnL8guGGfwG/5jmhM8znJLZ/Jr7uMMRykZZx9V4HCuD0q1iZQzB91Hb1q53iO5zV9jjgsZQo5qfVJxRAdKVvkMwAtkwowZ7gpqXHRZl/GEVbeGoY3KRChjeF9zRn06K5YS6rWqNwAkz4O3dqdE51gWC0bA9zKJryC15kAceTZ9L1dkNTe4FkwVn4LXQF/BuhrFrumWwo8aNzNz/l1hm/dZWWzJmycPajMvt0H3CJdxIEcitEnuBL15+vv8Pl2bYvMg17KnbwqOzr5vFGDBt5xVWOJT5APn5CYsCSyApuGVXWZWj0xbYzSayGyHAYV4lbrMLDQLUYbtOCC1
  magna087.ceph.redhat.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDL5XBcp/zElFBVLULh7GRt6hHp9nNGrQQewOIYmarg+SSqT0KRlPKTozG/rSBBH7iUPCwl/eMaWWr7cKVlVUMgn7JiLfQ2TsEDWBotHTZ/3Qal3W+YSrWmtIamUFnxmkB8OGt9Uu17s6ebWR/Oic/LOsQt9LlOdz69YX6tVEJne4xsLD0Fcwr8fOQdsAC9yBRw3mvnyLhcBWSCGX2jY7Do304w1dtycMGnjkdGsQCF1imKtXVgfRSfiTiLZID5GaSwZvCVoKsWpHveO0hTvOCq/4lwHO57k4eHgxFcCOsMJhpzuobVwrFgkS8bwW/XrPkttFDPYVOjReAXLjQ2OYkR
tasks:
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds: null
- rbd:
    all:
      image_size: 20480
- exec:
    client.0:
    - modprobe rbd
    - echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
    - echo 'module rbd +p' > /sys/kernel/debug/dynamic_debug/control
- workunit:
    clients:
      all:
      - suites/ffsb.sh

#7 Updated by Haomai Wang over 8 years ago

Does this problem make sense http://tracker.ceph.com/issues/12691

#8 Updated by Josh Durgin over 8 years ago

Looking at the osd logs, it seems like in one case at least this happened due to dup op detection having limited history.

The out of order op was 60415 detected on osd.1:

2015-09-08 20:59:04.991079 7f2d17a7e700 -1 osd.1 pg_epoch: 121 pg[0.12( v 120'2323 (0'0,120'2323] local-les=116 n=116 ec=1 l
es/c 116/100 115/115/115) [1,0] r=0 lpr=115 pi=93-114/3 rops=1 crt=111'2225 lcod 120'2322 mlcod 64'365 active+recovering+degraded] b
ad op order, already applied 60549 > this 60415

The primary for this object was osd.3. Op 60415 completed on osd.3, and was trimmed from its dup op log later.

The client completes more writes, up to tid 60549. Then it resends earlier writes that it presumable did not receive the response for yet, including 60415. Since the osd doesn't detect this as a dup, it proceeds to hit the out of order ops assert.

So this doesn't look like a bug in the client to me, just a consequence of an incredibly write-heavy workload, osd thrashing, and a bounded dup op detection scheme. I'm not sure there's much we can do about that.

#9 Updated by Josh Durgin over 8 years ago

Increasing osd_min_pg_log_entries and osd_max_pg_log_entries should avoid the problem. We should probably add that to the ffsb job in the krbd suite.

#10 Updated by Vasu Kulkarni over 8 years ago

Josh,

can you let me know what values I can try for osd_min_pg_log_entries and osd_max_pg_log_entries.

#11 Updated by Ilya Dryomov over 8 years ago

Is the dup op log that small? That's <200 tids we are talking about.
Josh, can you attach those logs? It'd be interesting to look at the timestamps.

#12 Updated by Josh Durgin over 8 years ago

That's a good point. Looking at the logs again, I mistook where the pg log trim happened - it was after the crash, so I don't think it's the culprit after all. Snippets related to the ops are attached, I can extract larger portions of those logs if you want, the whole logs are several gigs though.

#13 Updated by Josh Durgin over 8 years ago

#14 Updated by Ilya Dryomov almost 8 years ago

  • Category changed from rbd to libceph
  • Assignee set to Ilya Dryomov
  • Priority changed from Normal to Low

OSD client has been rewritten in 4.7.

#15 Updated by Ilya Dryomov about 5 years ago

  • Status changed from Need More Info to Resolved

Haven't seen out of order ops on 4.7 and later kernels.

Also available in: Atom PDF