Project

General

Profile

Bug #36388

osd: "out of order op"

Added by Patrick Donnelly over 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x84) [0x55bc98d98b2a]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x18e4) [0x55bc9907ff04]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3424) [0x55bc99083424]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 4: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd50) [0x55bc990857a0]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b3) [0x55bc98eb9cd3]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 6: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55bc991571d2]
2018-10-10T02:13:48.640 INFO:tasks.ceph.osd.2.smithi178.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6f2) [0x55bc98ed06a2]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x496) [0x55bc99513c16]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bc9951b3d0]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 10: (()+0x76db) [0x7f987f2446db]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 11: (clone()+0x3f) [0x7f987dfdf88f]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr:2018-10-10 02:13:48.633 7f9859741700 -1 /build/ceph-14.0.0-3932-g841b270/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9859741700 time 2018-10-10 02:13:48.636668
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr:/build/ceph-14.0.0-3932-g841b270/src/osd/PrimaryLogPG.cc: 4002: abort()
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr:
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: ceph version 14.0.0-3932-g841b270 (841b27044263faff77f3bed42b6fcb06b916d3a7) nautilus (dev)
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x84) [0x55bc98d98b2a]
2018-10-10T02:13:48.641 INFO:tasks.ceph.osd.2.smithi178.stderr: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x18e4) [0x55bc9907ff04]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3424) [0x55bc99083424]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 4: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd50) [0x55bc990857a0]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b3) [0x55bc98eb9cd3]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 6: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55bc991571d2]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6f2) [0x55bc98ed06a2]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x496) [0x55bc99513c16]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bc9951b3d0]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 10: (()+0x76db) [0x7f987f2446db]
2018-10-10T02:13:48.642 INFO:tasks.ceph.osd.2.smithi178.stderr: 11: (clone()+0x3f) [0x7f987dfdf88f]

From: /ceph/teuthology-archive/pdonnell-2018-10-09_01:01:26-kcephfs-wip-pdonnell-testing-20181008.224656-distro-basic-smithi/3118657/teuthology.log

Core: /ceph/teuthology-archive/pdonnell-2018-10-09_01:01:26-kcephfs-wip-pdonnell-testing-20181008.224656-distro-basic-smithi/3118657/remote/smithi178/coredump/1539137628.13591.core

Branch: https://github.com/ceph/ceph-ci/tree/wip-pdonnell-testing-20181008.224656


Related issues

Related to RADOS - Bug #24320: out of order reply and/or osd assert with set-chunks-read.yaml Resolved 05/26/2018

History

#1 Updated by Josh Durgin over 5 years ago

This looks like the dup op entries were exceeded so the op was not detected as a dup. Perhaps we should increase the dup ops for this suite/test.

#2 Updated by Sage Weil about 5 years ago

  • Related to Bug #24320: out of order reply and/or osd assert with set-chunks-read.yaml added

#3 Updated by Josh Durgin over 4 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF