Actions
Bug #16975
closed"ReplicatedPG.cc: 3030: FAILED assert(0 == "out of order op")" in rados-jewel-distro-basic-smithi
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados, upgrade/jewel-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2016-08-06_22:00:03-rados-jewel-distro-basic-smithi/
Job: 352796
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-08-06_22:00:03-rados-jewel-distro-basic-smithi/352796/teuthology.log
2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:01.913767 7f11ff4ef800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:01.965864 7f11ff4ef800 -1 osd.0 113 log_to_monitors {default=true} 2016-08-08T15:52:06.898 INFO:tasks.ceph.osd.0.smithi012.stderr:2016-08-08 22:52:06.600435 7f11dc2a4700 -1 osd.0 pg_epoch: 143 pg[6.3( v 143'7654 (143'7500,143'7654] local-les=141 n=12 ec=12 les/c/f 141/141/0 140/140/140) [0,5] r=0 lpr=140 luod=143'7636 crt=143'7553 lcod 143'7635 mlcod 143'7635 active+clean] bad op order, already applied 30367 > this 29599 2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f11dc2a4700 time 2016-08-08 22:52:06.600452 2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr:osd/ReplicatedPG.cc: 3030: FAILED assert(0 == "out of order op") 2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr: ceph version 10.2.2-195-g5c98730 (5c98730854f11b0efb3b3e03be426ce2b7a999af) 2016-08-08T15:52:06.899 INFO:tasks.ceph.osd.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f11fff23645] 2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1a84) [0x7f11ffa23bb4] 2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 3: (ReplicatedPG::do_op(std::shared_ptr<OpRequest>&)+0x2727) [0x7f11ffa26877] 2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 4: (ReplicatedPG::do_request(std::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x747) [0x7f11ff9e2d27] 2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x41d) [0x7f11ff89779d] 2016-08-08T15:52:06.900 INFO:tasks.ceph.osd.0.smithi012.stderr: 6: (PGQueueable::RunVis::operator()(std::shared_ptr<OpRequest>&)+0x6d) [0x7f11ff8979ed] 2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x869) [0x7f11ff89c519] 2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x887) [0x7f11fff135e7] 2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f11fff15550] 2016-08-08T15:52:06.901 INFO:tasks.ceph.osd.0.smithi012.stderr: 10: (()+0x7dc5) [0x7f11fdc3fdc5] 2016-08-08T15:52:06.902 INFO:tasks.ceph.osd.0.smithi012.stderr: 11: (clone()+0x6d) [0x7f11fc2cb28d] 2016-08-08T15:52:06.902 INFO:tasks.ceph.osd.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Yuri Weinstein over 7 years ago
- Related to Bug #15407: "ReplicatedPG.cc: 2986: FAILED assert(0 == "out of order op")" with short_pg_log added
Updated by Yuri Weinstein over 7 years ago
Also in upgrades
Run: http://pulpito.ceph.com/teuthology-2016-08-10_04:20:03-upgrade:jewel-x-master-distro-basic-vps/
Jobs: ['357375', '357381']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-08-10_04:20:03-upgrade:jewel-x-master-distro-basic-vps/357375/teuthology.log
2016-08-10T14:59:42.354 INFO:tasks.ceph.osd.0.vpm063.stderr:/srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f5f9ad7f700 time 2016-08-10 14:59:41.643075 2016-08-10T14:59:42.354 INFO:tasks.ceph.osd.0.vpm063.stderr:/srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/osd/ReplicatedPG.cc: 3045: FAILED assert(0 == "out of order op") 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: ceph version v11.0.0-1386-g0d41383 (0d41383887b728c51d4b53eb2371cb48aba66107) 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xd8c7c5] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x2204) [0x8fbc84] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 3: (ReplicatedPG::do_op(std::shared_ptr<OpRequest>&)+0x2810) [0x8fe590] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 4: (ReplicatedPG::do_request(std::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x777) [0x8b9b77] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x41d) [0x768bbd] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 6: (PGQueueable::RunVis::operator()(std::shared_ptr<OpRequest> const&)+0x6d) [0x768e0d] 2016-08-10T14:59:42.355 INFO:tasks.ceph.osd.0.vpm063.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x86c) [0x78a42c] 2016-08-10T14:59:42.356 INFO:tasks.ceph.osd.0.vpm063.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x947) [0xd92437] 2016-08-10T14:59:42.356 INFO:tasks.ceph.osd.0.vpm063.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xd94590] 2016-08-10T14:59:42.356 INFO:tasks.ceph.osd.0.vpm063.stderr: 10: (()+0x7dc5) [0x7f5fb9218dc5] 2016-08-10T14:59:42.356 INFO:tasks.ceph.osd.0.vpm063.stderr: 11: (clone()+0x6d) [0x7f5fb80ffced] 2016-08-10T14:59:42.356 INFO:tasks.ceph.osd.0.vpm063.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this
Updated by Josh Durgin over 7 years ago
- Status changed from New to Fix Under Review
Updated by Josh Durgin over 7 years ago
- Status changed from Fix Under Review to Resolved
Actions