Project

General

Profile

Actions

Bug #25198

closed

FAILED assert(trim_to <= info.last_complete) in PGLog::trim()

Added by Neha Ojha over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2018-07-30T06:53:39.137 INFO:tasks.ceph.osd.1.smithi037.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-1624-g61cd0f9/rpm/el7/BUILD/ceph-14.0.0-1624-g61cd0f9/src/osd/PGLog.cc: In function 'void PGLog::trim(eversion_t, pg_info_t&, bool, bool)' thread 7f58208e7700 time 2018-07-30 06:53:39.145191
2018-07-30T06:53:39.137 INFO:tasks.ceph.osd.1.smithi037.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-1624-g61cd0f9/rpm/el7/BUILD/ceph-14.0.0-1624-g61cd0f9/src/osd/PGLog.cc: 179: FAILED assert(trim_to <= info.last_complete)
2018-07-30T06:53:39.141 INFO:tasks.ceph.osd.1.smithi037.stderr: ceph version 14.0.0-1624-g61cd0f9 (61cd0f963500449aea2f497b6e68b2fde0fec701) nautilus (dev)
2018-07-30T06:53:39.141 INFO:tasks.ceph.osd.1.smithi037.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564791de16db]
2018-07-30T06:53:39.141 INFO:tasks.ceph.osd.1.smithi037.stderr: 2: (()+0xab5897) [0x564791de1897]
2018-07-30T06:53:39.141 INFO:tasks.ceph.osd.1.smithi037.stderr: 3: (()+0x639c86) [0x564791965c86]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 4: (PG::RecoveryState::ReplicaActive::react(MTrim const&)+0x3f) [0x5647918d781f]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 5: (boost::statechart::simple_state<PG::RecoveryState::ReplicaActive, PG::RecoveryState::Started, PG::RecoveryState::RepNotRecovering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x254) [0x56479195eab4]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 6: (boost::statechart::simple_state<PG::RecoveryState::RepNotRecovering, PG::RecoveryState::ReplicaActive, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x123) [0x56479195c2d3]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0x564791939f8b]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 8: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x143) [0x56479191b833]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xcf) [0x5647918609cf]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 10: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564791ac00c0]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x52b) [0x564791874ecb]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d6) [0x564791de60d6]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564791de6cd0]
2018-07-30T06:53:39.142 INFO:tasks.ceph.osd.1.smithi037.stderr: 14: (()+0x7dd5) [0x7f5845cb1dd5]
2018-07-30T06:53:39.143 INFO:tasks.ceph.osd.1.smithi037.stderr: 15: (clone()+0x6d) [0x7f5844760b3d]
2018-07-30T06:53:39.143 INFO:tasks.ceph.osd.1.smithi037.stderr:*** Caught signal (Aborted) **

http://qa-proxy.ceph.com/teuthology/xxg-2018-07-30_05:25:06-rados-wip-hb-peers-distro-basic-smithi/2837916/teuthology.log


Related issues 3 (0 open3 closed)

Related to RADOS - Feature #23979: Limit pg log length during recovery/backfill so that we don't run out of memory.ResolvedNeha Ojha05/02/2018

Actions
Copied to RADOS - Backport #25199: luminous: FAILED assert(trim_to <= info.last_complete) in PGLog::trim()ResolvedNeha OjhaActions
Copied to RADOS - Backport #25200: mimic: FAILED assert(trim_to <= info.last_complete) in PGLog::trim()ResolvedNeha OjhaActions
Actions #1

Updated by Neha Ojha over 5 years ago

  • Related to Feature #23979: Limit pg log length during recovery/backfill so that we don't run out of memory. added
Actions #2

Updated by Neha Ojha over 5 years ago

  • Status changed from New to Fix Under Review
Actions #3

Updated by Sage Weil over 5 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #4

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #25199: luminous: FAILED assert(trim_to <= info.last_complete) in PGLog::trim() added
Actions #5

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #25200: mimic: FAILED assert(trim_to <= info.last_complete) in PGLog::trim() added
Actions #6

Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF