Bug #21310
Updated by Sage Weil over 6 years ago
To follow up on http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020555.html Please see this log archive (too big to attach): https://drive.google.com/open?id=0B0bD14x88J_pSndpVXdsWTVLVDA <pre> Sep 08 12:03:31 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: Started Ceph object storage daemon osd.16. Sep 08 12:03:31 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: starting osd.16 at - osd_data /var/lib/ceph/osd/ceph-16 /var/lib/ceph/osd/ceph-16/journal Sep 08 12:03:35 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:35.825537 7f8a7cd85d00 -1 osd.16 4774 log_to_monitors {default=true} Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825 Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:41.070467 7f8a638f4700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825 Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: -44> 2017-09-08 12:03:41.070467 7f8a638f4700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7f8a638f4700 time 2017-09-08 12:03:41.062825 Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/osd/PG.h: 467: FAILED assert(i->second.need == j->second.need) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f8a7d802510] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: *** Caught signal (Aborted) ** Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2017-09-08 12:03:41.123191 7f8a638f4700 -1 *** Caught signal (Aborted) ** Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 0> 2017-09-08 12:03:41.123191 7f8a638f4700 -1 *** Caught signal (Aborted) ** Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: in thread 7f8a638f4700 thread_name:tp_peering Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 1: (()+0xa23b21) [0x7f8a7d7c3b21] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 2: (()+0xf370) [0x7f8a7a3cf370] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 3: (gsignal()+0x37) [0x7f8a793f91d7] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 4: (abort()+0x148) [0x7f8a793fa8c8] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7f8a7d802684] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0xd7) [0x7f8a7d382427] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x190c) [0x7f8a7d34110c] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x326) [0x7f8a7d342086] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 9: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0xbb) [0x7f8a7d39f08b] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 10: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x15a) [0x7f8a7d39f4ca] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 11: (boost::statechart::simple_state<PG::RecoveryState::WaitUpThru, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd0) [0x7f8a7d39bdf0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x6b) [0x7f8a7d37bebb] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 13: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x91) [0x7f8a7d37c011] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 14: (PG::handle_activate_map(PG::RecoveryCtx*)+0x131) [0x7f8a7d3485d1] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 15: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x72f) [0x7f8a7d2936bf] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 16: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x173) [0x7f8a7d293cf3] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 17: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x17) [0x7f8a7d2f5087] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x7f8a7d8090fe] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 19: (ThreadPool::WorkThread::entry()+0x10) [0x7f8a7d809fe0] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 20: (()+0x7dc5) [0x7f8a7a3c7dc5] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: 21: (clone()+0x6d) [0x7f8a794bb76d] Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs ceph-osd[20853]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: ceph-osd@16.service: main process exited, code=killed, status=6/ABRT Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: Unit ceph-osd@16.service entered failed state. Sep 08 12:03:41 r72-k7-06-01.k8s.ash1.cloudsys.tmcs systemd[1]: ceph-osd@16.service failed. </pre>