Bug #36388
osd: "out of order op"
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x84) [0x55bc98d98b2a] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x18e4) [0x55bc9907ff04] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3424) [0x55bc99083424] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 4: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd50) [0x55bc990857a0] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b3) [0x55bc98eb9cd3] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 6: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55bc991571d2] 2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6f2) [0x55bc98ed06a2] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x496) [0x55bc99513c16] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bc9951b3d0] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 10: (()+0x76db) [0x7f987f2446db] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 11: (clone()+0x3f) [0x7f987dfdf88f] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr:2018-10-10 02:13:48.633 7f9859741700 -1 /build/ceph-14.0.0-3932-g841b270/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9859741700 time 2018-10-10 02:13:48.636668 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr:/build/ceph-14.0.0-3932-g841b270/src/osd/PrimaryLogPG.cc: 4002: abort() 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: ceph version 14.0.0-3932-g841b270 (841b27044263faff77f3bed42b6fcb06b916d3a7) nautilus (dev) 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x84) [0x55bc98d98b2a] 2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x18e4) [0x55bc9907ff04] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3424) [0x55bc99083424] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 4: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd50) [0x55bc990857a0] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b3) [0x55bc98eb9cd3] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 6: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55bc991571d2] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6f2) [0x55bc98ed06a2] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x496) [0x55bc99513c16] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bc9951b3d0] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 10: (()+0x76db) [0x7f987f2446db] 2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 11: (clone()+0x3f) [0x7f987dfdf88f]
From: /ceph/teuthology-archive/pdonnell-2018-10-09_01:01:26-kcephfs-wip-pdonnell-testing-20181008.224656-distro-basic-smithi/3118657/teuthology.log
Core: /ceph/teuthology-archive/pdonnell-2018-10-09_01:01:26-kcephfs-wip-pdonnell-testing-20181008.224656-distro-basic-smithi/3118657/remote/smithi178/coredump/1539137628.13591.core
Branch: https://github.com/ceph/ceph-ci/tree/wip-pdonnell-testing-20181008.224656
Related issues
History
#1 Updated by Josh Durgin over 5 years ago
This looks like the dup op entries were exceeded so the op was not detected as a dup. Perhaps we should increase the dup ops for this suite/test.
#2 Updated by Sage Weil about 5 years ago
- Related to Bug #24320: out of order reply and/or osd assert with set-chunks-read.yaml added
#3 Updated by Josh Durgin over 4 years ago
- Status changed from New to Resolved