Project

General

Profile

Bug #21310

src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need) (bluestore+ec+rbd)

Added by Jean-Francois Nadeau about 2 years ago. Updated almost 2 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
Start date:
09/08/2017
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

To follow up on http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020555.html

Please see this log archive (too big to attach): https://drive.google.com/open?id=0B0bD14x88J_pSndpVXdsWTVLVDA

Sep 08 12:03:31 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: Started Ceph object storage daemon osd.16.
Sep 08 12:03:31 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: starting osd.16 at - osd_data /var/lib/ceph/osd/ceph-16 /var/lib/ceph/osd/ceph-16/journal
Sep 08 12:03:35 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:35.825537 7f8a7cd85d00 -1 osd.16 4774 log_to_monitors {default=true}
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:41.070467 7f8a638f4700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: -44> 2017-09-08 12:03:41.070467 7f8a638f4700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: *** Caught signal (Aborted) **
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:41.123191 7f8a638f4700 -1 *** Caught signal (Aborted) **
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 0> 2017-09-08 12:03:41.123191 7f8a638f4700 -1 *** Caught signal (Aborted) **
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d]
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: ceph-osd@16.service: main process exited, code=killed, status=6/ABRT
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: Unit ceph-osd@16.service entered failed state.
Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: ceph-osd@16.service failed.

History

#1 Updated by Sage Weil almost 2 years ago

  • Description updated (diff)

#2 Updated by Sage Weil almost 2 years ago

  • Subject changed from Multiple OSD crashing on 12.2.0. Bluestore / EC pool / rbd to src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need) (bluestore+ec+rbd)

#3 Updated by Sage Weil almost 2 years ago

  • Status changed from New to Need More Info

The log in the bug above doesn't seem to match what is in the email thread.. which refers to #20222 (which is almost certainly a dup of #21171, now fixed).

Can you clarify whether this is a separate issue? Also, the log linked doesn't seem to have any crashes in it. Thanks!

#4 Updated by Jean-Francois Nadeau almost 2 years ago

Apologies if they are different issues... both occurred with rbd bench on EC pools.

If at least one is a dup of #21171 do we have a daily test build/RPMs I can use to confirm the fix or the remaining issue (I dont see new RPMs under https://download.ceph.com/rpm-luminous/el7/x86_64/) ?

#5 Updated by Jean-Francois Nadeau almost 2 years ago

After updating to 12.2.1 and running several rbd bench runs in the same context (rbd+EC), I could not reproduce the issue. Thanks for the support.

Also available in: Atom PDF