Bug #14008: "FAILED assert(0 == "out of order op")" in rbd-hammer-distro-basic-openstack - Ceph - Ceph

Actions

Copy link

Bug #14008

closed

"FAILED assert(0 == "out of order op")" in rbd-hammer-distro-basic-openstack

Added by Yuri Weinstein over 8 years ago. Updated about 8 years ago.

Status:

Can't reproduce

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

rados

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2015-12-06_18:00:02-rbd-hammer-distro-basic-openstack/
Job: ['29969']
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2015-12-06_18:00:02-rbd-hammer-distro-basic-openstack/29969/teuthology.log

2015-12-06T18:52:49.883 INFO:tasks.ceph.osd.3.target066241.stderr:2015-12-06 18:52:49.781079 7f6ce763e700 -1 osd.3 pg_epoch: 7 pg[0.8( v 7'831 (0'0,7'831] local-les=5 n=91 ec=1 les/c 5/5 4/4/4) [3,5] r=0 lpr=4 crt=7'829 lcod 7'830 mlcod 7'830 active+clean] bad op order, already applied 19269 > this 19264
2015-12-06T18:52:49.886 INFO:tasks.ceph.osd.3.target066241.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)' thread 7f6ce763e700 time 2015-12-06 18:52:49.781092
2015-12-06T18:52:49.886 INFO:tasks.ceph.osd.3.target066241.stderr:osd/ReplicatedPG.cc: 2354: FAILED assert(0 == "out of order op")
2015-12-06T18:52:49.886 INFO:tasks.ceph.osd.3.target066241.stderr: ceph version 0.94.5-173-gd064636 (d06463604cb8daeda288e824e8812352c0d6a7d9)
2015-12-06T18:52:49.887 INFO:tasks.ceph.osd.3.target066241.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc5a9b]
2015-12-06T18:52:49.887 INFO:tasks.ceph.osd.3.target066241.stderr: 2: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0x1766) [0x8a59a6]
2015-12-06T18:52:49.888 INFO:tasks.ceph.osd.3.target066241.stderr: 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x4559) [0x8aa559]
2015-12-06T18:52:49.888 INFO:tasks.ceph.osd.3.target066241.stderr: 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x8444ba]
2015-12-06T18:52:49.888 INFO:tasks.ceph.osd.3.target066241.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3bd) [0x6a1aed]
2015-12-06T18:52:49.889 INFO:tasks.ceph.osd.3.target066241.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x331) [0x6a2011]
2015-12-06T18:52:49.889 INFO:tasks.ceph.osd.3.target066241.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0xbb55ef]
2015-12-06T18:52:49.889 INFO:tasks.ceph.osd.3.target066241.stderr: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xbb7710]
2015-12-06T18:52:49.889 INFO:tasks.ceph.osd.3.target066241.stderr: 9: (()+0x8182) [0x7f6d066f6182]
2015-12-06T18:52:49.889 INFO:tasks.ceph.osd.3.target066241.stderr: 10: (clone()+0x6d) [0x7f6d04c6147d]
2015-12-06T18:52:49.890 INFO:tasks.ceph.osd.3.target066241.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.