Ceph

Run: http://pulpito.ceph.com/teuthology-2016-08-06_22:00:03-rados-jewel-distro-basic-smithi/
Job: 352796
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-08-06_22:00:03-rados-jewel-distro-basic-smithi/352796/teuthology.log

2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:01.913767 7f11ff4ef800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:01.965864 7f11ff4ef800 -1 osd.0 113 log_to_monitors {default=true}
2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:06.600435 7f11dc2a4700 -1 osd.0 pg_epoch: 143 pg[6.3( v 143'7654 (143'7500,143'7654] local-les=141 n=12 ec=12 les/c/f 141/141/0 140/140/140) [0,5] r=0 lpr=140 luod=143'7636 crt=143'7553 lcod 143'7635 mlcod 143'7635 active+clean] bad op order, already applied 30367 > this 29599
2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f11dc2a4700 time 2016-08-08 22:52:06.600452
2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr:osd/ReplicatedPG.cc: 3030: FAILED assert(0 == "out of order op")
2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr: ceph version 10.2.2-195-g5c98730 (5c98730854f11b0efb3b3e03be426ce2b7a999af)
2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f11fff23645]
2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1a84) [0x7f11ffa23bb4]
2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 3: (ReplicatedPG::do_op(std::shared_ptr<OpRequest>&)+0x2727) [0x7f11ffa26877]
2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 4: (ReplicatedPG::do_request(std::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x747) [0x7f11ff9e2d27]
2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x41d) [0x7f11ff89779d]
2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 6: (PGQueueable::RunVis::operator()(std::shared_ptr<OpRequest>&)+0x6d) [0x7f11ff8979ed]
2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x869) [0x7f11ff89c519]
2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x887) [0x7f11fff135e7]
2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f11fff15550]
2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 10: (()+0x7dc5) [0x7f11fdc3fdc5]
2016-08-08T15:52:06.902 INFO:tasks.ceph.osd.0.smithi012.stderr: 11: (clone()+0x6d) [0x7f11fc2cb28d]
2016-08-08T15:52:06.902 INFO:tasks.ceph.osd.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.