Project

General

Profile

Bug #15418

"ReplicatedPG.cc: 8765: FAILED assert(object_contexts.empty())" in rados-hammer-distro-basic-vps

Added by Yuri Weinstein almost 8 years ago. Updated over 7 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2016-04-06_09:00:02-rados-hammer-distro-basic-vps/
Job: 111471
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-04-06_09:00:02-rados-hammer-distro-basic-vps/111471/teuthology.log

2016-04-06T10:44:53.842 INFO:teuthology.orchestra.run.vpm009:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2016-04-06T10:44:54.059 INFO:tasks.ceph.osd.0.vpm009.stderr:osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::on_flushed()' thread 7f82b79b2700 time 2016-04-06 17:44:53.493439
2016-04-06T10:44:54.060 INFO:tasks.ceph.osd.0.vpm009.stderr:osd/ReplicatedPG.cc: 8765: FAILED assert(object_contexts.empty())
2016-04-06T10:44:54.306 INFO:tasks.ceph.osd.0.vpm009.stderr: ceph version 0.94.6-206-g77fbf58 (77fbf581cb2259146938a737c299d6cf762303d1)
2016-04-06T10:44:54.306 INFO:tasks.ceph.osd.0.vpm009.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbae2ab]
2016-04-06T10:44:54.306 INFO:tasks.ceph.osd.0.vpm009.stderr: 2: (ReplicatedPG::on_flushed()+0x2d1) [0x82a2b1]
2016-04-06T10:44:54.307 INFO:tasks.ceph.osd.0.vpm009.stderr: 3: (PG::RecoveryState::Started::react(PG::FlushedEvt const&)+0x3f) [0x7a765f]
2016-04-06T10:44:54.307 INFO:tasks.ceph.osd.0.vpm009.stderr: 4: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f0) [0x8202b0]
2016-04-06T10:44:54.307 INFO:tasks.ceph.osd.0.vpm009.stderr: 5: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1a9) [0x81d5d9]
2016-04-06T10:44:54.307 INFO:tasks.ceph.osd.0.vpm009.stderr: 6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x808b2b]
2016-04-06T10:44:54.307 INFO:tasks.ceph.osd.0.vpm009.stderr: 7: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x1ce) [0x7b5a6e]
2016-04-06T10:44:54.308 INFO:tasks.ceph.osd.0.vpm009.stderr: 8: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x2c0) [0x6a86c0]
2016-04-06T10:44:54.308 INFO:tasks.ceph.osd.0.vpm009.stderr: 9: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x18) [0x7012f8]
2016-04-06T10:44:54.308 INFO:tasks.ceph.osd.0.vpm009.stderr: 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xb9ee96]
2016-04-06T10:44:54.309 INFO:tasks.ceph.osd.0.vpm009.stderr: 11: (ThreadPool::WorkThread::entry()+0x10) [0xb9ff40]
2016-04-06T10:44:54.309 INFO:tasks.ceph.osd.0.vpm009.stderr: 12: (()+0x8182) [0x7f82d0850182]
2016-04-06T10:44:54.309 INFO:tasks.ceph.osd.0.vpm009.stderr: 13: (clone()+0x6d) [0x7f82cedbb47d]
2016-04-06T10:44:54.309 INFO:tasks.ceph.osd.0.vpm009.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

History

#1 Updated by Yuri Weinstein almost 8 years ago

Run: http://pulpito.ceph.com/teuthology-2016-04-07_22:00:02-rados-jewel-distro-basic-smithi/
Job: 115412
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-04-07_22:00:02-rados-jewel-distro-basic-smithi/115412/teuthology.log

2016-04-09T04:33:52.219 INFO:tasks.ceph.osd.0.smithi039.stderr:osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::on_flushed()' thread 7fc9ab920700 time 2016-04-09 11:33:52.215716
2016-04-09T04:33:52.219 INFO:tasks.ceph.osd.0.smithi039.stderr:osd/ReplicatedPG.cc: 9925: FAILED assert(object_contexts.empty())
2016-04-09T04:33:52.220 INFO:tasks.ceph.osd.0.smithi039.stderr: ceph version 10.1.1 (ce50389b773fe7f72fca40a3dd69cfe6613eaeb1)
2016-04-09T04:33:52.220 INFO:tasks.ceph.osd.0.smithi039.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fc9c4b43f4b]
2016-04-09T04:33:52.220 INFO:tasks.ceph.osd.0.smithi039.stderr: 2: (ReplicatedPG::get_watchers(std::list<obj_watch_item_t, std::allocator<obj_watch_item_t> >&)+0) [0x7fc9c46088c0]
2016-04-09T04:33:52.220 INFO:tasks.ceph.osd.0.smithi039.stderr: 3: (PG::RecoveryState::Started::react(PG::FlushedEvt const&)+0x3f) [0x7fc9c458133f]
2016-04-09T04:33:52.221 INFO:tasks.ceph.osd.0.smithi039.stderr: 4: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f0) [0x7fc9c45fd080]
2016-04-09T04:33:52.221 INFO:tasks.ceph.osd.0.smithi039.stderr: 5: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xc0) [0x7fc9c45fea40]
2016-04-09T04:33:52.221 INFO:tasks.ceph.osd.0.smithi039.stderr: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xc0) [0x7fc9c45fc9f0]
2016-04-09T04:33:52.221 INFO:tasks.ceph.osd.0.smithi039.stderr: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xbc) [0x7fc9c45fa23c]
2016-04-09T04:33:52.222 INFO:tasks.ceph.osd.0.smithi039.stderr: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x7fc9c45e472b]
2016-04-09T04:33:52.222 INFO:tasks.ceph.osd.0.smithi039.stderr: 9: (PG::handle_peering_event(std::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x1d5) [0x7fc9c45acdf5]
2016-04-09T04:33:52.222 INFO:tasks.ceph.osd.0.smithi039.stderr: 10: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x249) [0x7fc9c450cf89]
2016-04-09T04:33:52.222 INFO:tasks.ceph.osd.0.smithi039.stderr: 11: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x7fc9c4555162]
2016-04-09T04:33:52.223 INFO:tasks.ceph.osd.0.smithi039.stderr: 12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0x7fc9c4b354ae]
2016-04-09T04:33:52.223 INFO:tasks.ceph.osd.0.smithi039.stderr: 13: (ThreadPool::WorkThread::entry()+0x10) [0x7fc9c4b36390]
2016-04-09T04:33:52.223 INFO:tasks.ceph.osd.0.smithi039.stderr: 14: (()+0x8182) [0x7fc9c2e07182]
2016-04-09T04:33:52.223 INFO:tasks.ceph.osd.0.smithi039.stderr: 15: (clone()+0x6d) [0x7fc9c0f3547d]
2016-04-09T04:33:52.224 INFO:tasks.ceph.osd.0.smithi039.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

#2 Updated by Yuri Weinstein almost 8 years ago

  • Priority changed from Normal to Urgent
  • Release set to jewel

#3 Updated by Sage Weil almost 8 years ago

moved second trace to #15505

#4 Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF