Bug #48850: "FAILED ceph_assert(m_pending_ops == 0)" in TestImageReplayer/3.SnapshotUnprotect - rbd - Ceph

Actions

Copy link

Bug #48850

closed

"FAILED ceph_assert(m_pending_ops == 0)" in TestImageReplayer/3.SnapshotUnprotect

Added by Jason Dillaman over 3 years ago. Updated about 2 years ago.

Status:

Duplicate

Priority:

High

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description


[ RUN      ] TestImageReplayer/3.SnapshotUnprotect
flushed
remote_snap_id=6
local_snap_id=5, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=531e9485-da5f-4764-bf97-cf302bdce64a, primary_snap_id=6, last_copied_object_number=0, snap_seqs={5=4,6=18446744073709551614}]
flushed
remote_snap_id=7
local_snap_id=6, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=531e9485-da5f-4764-bf97-cf302bdce64a, primary_snap_id=7, last_copied_object_number=0, snap_seqs={5=4,7=18446744073709551614}]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f3597ff7700 time 2021-01-12T07:08:31.392685+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)
 ceph version Development (no_version) pacific (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19d) [0x7f36591baf1b]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x167419d) [0x7f36591bb19d]
 3: (AsyncOpTracker::~AsyncOpTracker()+0x45) [0x7f36590c50d9]
 4: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x68) [0x56250c84836c]
 5: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x18) [0x56250c848404]
 6: (librbd::ImageCtx::shutdown()+0x2b) [0x56250c8177f9]
 7: (librbd::image::CloseRequest<librbd::ImageCtx>::finish()+0x1b) [0x56250c9d7915]
 8: (librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher(int)+0x271) [0x56250c9d7893]
 9: (librbd::util::detail::C_CallbackAdapter<librbd::image::CloseRequest<librbd::ImageCtx>, &librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher>::finish(int)+0x24) [0x56250c9d87d0]
 10: (Context::complete(int)+0x27) [0x56250c008df5]
 11: (librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}::operator()() const+0x34) [0x56250c008ea6]
 12: (void boost::asio::asio_handler_invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, ...)+0x68) [0x56250c035c67]
 13: (void boost_asio_handler_invoke_helpers::invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, {lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, {lambda()#1}&)+0x33) [0x56250c0355fc]
 14: (void boost::asio::system_executor::dispatch<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, std::allocator<void> >(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&&, std::allocator<void> const&) const+0x51) [0x56250c034cf1]
 15: (boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>::operator()()+0x52) [0x56250c034a82]
 16: (void boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >(boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>&, ...)+0x68) [0x56250c034646]
 17: (void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>&, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&)+0x33) [0x56250c033e12]
 18: (void boost::asio::detail::handler_work<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>, boost::asio::system_executor, boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >::complete<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&)+0x27) [0x56250c03238b]
 19: (boost::asio::detail::completion_handler<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xd6) [0x56250c02fdac]
 20: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x36) [0x56250c809ab6]
 21: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xa2) [0x56250c80a254]
 22: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x36) [0x56250c809ab6]
 23: (boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)+0x1c2) [0x56250cbb14b6]
 24: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0xf4) [0x56250cbb108a]
 25: (boost::asio::io_context::run()+0x48) [0x56250cf3e138]
 26: (ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const+0x1f) [0x56250cf3e76f]
 27: (void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x56250cf4394e]
 28: (std::__invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x20) [0x56250cf42e1e]
 29: (std::invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x20) [0x56250cf41e86]
 30: _ZZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_ENKUlSA_SD_E_clIS3_JEEEDaSA_SD_()
 31: (std::basic_string_view<char, std::char_traits<char> > std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&, {lambda()#1}&&)+0x37) [0x56250cf45f80]
 32: (std::__invoke_result<std::basic_string_view<char, std::char_traits<char> >, (make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)...>::type std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}&&)+0x37) [0x56250cf45ceb]
 33: (void std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)+0x43) [0x56250cf45ab7]
 34: (std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::operator()()+0x18) [0x56250cf457d6]
 35: (std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> > >::_M_run()+0x1c) [0x56250cf45162]
 36: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd0d80) [0x7f3656f53d80]
 37: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f36632256db]
 38: clone()

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Neha Ojha about 3 years ago

Backport set to pacific

similar

[ RUN      ] TestImageReplayer/4.SnapshotLimit
remote_snap_id=4
local_snap_id=3, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3901c24-38d3-49cb-8647-8a276cb07dbd, primary_snap_id=4, last_copied_object_number=0, snap_seqs={4=18446744073709551614}]
flushed
remote_snap_id=5
local_snap_id=4, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3901c24-38d3-49cb-8647-8a276cb07dbd, primary_snap_id=5, last_copied_object_number=0, snap_seqs={5=18446744073709551614}]
flushed
remote_snap_id=6
local_snap_id=5, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3901c24-38d3-49cb-8647-8a276cb07dbd, primary_snap_id=6, last_copied_object_number=0, snap_seqs={6=18446744073709551614}]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7fe5017fa700 time 2021-02-04T00:38:56.538295+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)
 ceph version Development (no_version) pacific (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19d) [0x7fe5c2bccb6d]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x166cdef) [0x7fe5c2bccdef]
 3: (AsyncOpTracker::~AsyncOpTracker()+0x45) [0x7fe5c2ad6db1]
 4: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x68) [0x55fc08fcdbec]
 5: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x18) [0x55fc08fcdc84]
 6: (librbd::ImageCtx::shutdown()+0x2b) [0x55fc08f9d079]
 7: (librbd::image::CloseRequest<librbd::ImageCtx>::finish()+0x1b) [0x55fc0915d195]
 8: (librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher(int)+0x271) [0x55fc0915d113]
 9: (librbd::util::detail::C_CallbackAdapter<librbd::image::CloseRequest<librbd::ImageCtx>, &librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher>::finish(int)+0x24) [0x55fc0915e050]
 10: (Context::complete(int)+0x27) [0x55fc0878e523]
 11: (librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}::operator()() const+0x34) [0x55fc0878e5d4]
 12: (void boost::asio::asio_handler_invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, ...)+0x68) [0x55fc087bb48d]
 13: (void boost_asio_handler_invoke_helpers::invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, {lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, {lambda()#1}&)+0x33) [0x55fc087bae22]
 14: (void boost::asio::system_executor::dispatch<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, std::allocator<void> >(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&&, std::allocator<void> const&) const+0x51) [0x55fc087ba517]
 15: (boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>::operator()()+0x52) [0x55fc087ba2a8]
 16: (void boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >(boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>&, ...)+0x68) [0x55fc087b9e6c]
 17: (void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>&, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&)+0x33) [0x55fc087b9638]
 18: (void boost::asio::detail::handler_work<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>, boost::asio::system_executor, boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >::complete<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&)+0x27) [0x55fc087b7bb1]
 19: (boost::asio::detail::completion_handler<boost::asio::detail::work_dispatcher<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xd6) [0x55fc087b55d2]
 20: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x36) [0x55fc08f8f336]
 21: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xa2) [0x55fc08f8fad4]
 22: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x36) [0x55fc08f8f336]
 23: (boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)+0x1c2) [0x55fc09336d74]
 24: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0xf4) [0x55fc09336948]
 25: (boost::asio::io_context::run()+0x48) [0x55fc096d1080]
 26: (ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const+0x1f) [0x55fc096d16b7]
 27: (void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x55fc096d6896]
 28: (std::__invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x20) [0x55fc096d5d66]
 29: (std::invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x20) [0x55fc096d4dce]
 30: _ZZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_ENKUlSA_SD_E_clIS3_JEEEDaSA_SD_()
 31: (std::basic_string_view<char, std::char_traits<char> > std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&, {lambda()#1}&&)+0x37) [0x55fc096d8ec8]
 32: (std::__invoke_result<std::basic_string_view<char, std::char_traits<char> >, (make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)...>::type std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}&&)+0x37) [0x55fc096d8c33]
 33: (void std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)+0x43) [0x55fc096d89ff]
 34: (std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::operator()()+0x18) [0x55fc096d871e]
 35: (std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> > >::_M_run()+0x1c) [0x55fc096d80aa]
 36: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd0d80) [0x7fe5c096cd80]
 37: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fe5ccc346db]
 38: clone()

https://jenkins.ceph.com/job/ceph-pull-requests/68629/consoleFull#-19117738427725d52c-2930-43cb-b77a-ae0a919c2170 - https://github.com/ceph/ceph/pull/39284

Actions

Copy link

Updated by Jason Dillaman about 3 years ago

Priority changed from Normal to High

Actions

Copy link

Updated by Neha Ojha about 3 years ago

https://jenkins.ceph.com/job/ceph-pull-requests/72169/consoleFull#12012256367725d52c-2930-43cb-b77a-ae0a919c2170

Actions

Copy link

Updated by Kefu Chai almost 3 years ago

[ RUN      ] TestImageReplayer/3.SnapshotUnprotect
flushed
remote_snap_id=6
local_snap_id=5, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=78790c31-91bd-4388-8eb7-8bb80beabd86, primary_snap_id=6, last_copied_object_number=0, snap_seqs={5=4,6=18446744073709551614}]
flushed
remote_snap_id=7
local_snap_id=6, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=78790c31-91bd-4388-8eb7-8bb80beabd86, primary_snap_id=7, last_copied_object_number=0, snap_seqs={5=4,7=18446744073709551614}]
../src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f580bff7700 time 2021-04-27T10:32:57.479026+0000
../src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)
 ceph version Development (no_version) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a1) [0x7f58c8011ead]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x169714b) [0x7f58c801214b]
 3: (AsyncOpTracker::~AsyncOpTracker()+0x49) [0x7f58c7f1133b]
 4: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x6c) [0x55aa21235f78]
 5: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x1c) [0x55aa21235ff0]
 6: (librbd::ImageCtx::shutdown()+0x2f) [0x55aa21203da9]
 7: (librbd::image::CloseRequest<librbd::ImageCtx>::finish()+0x1f) [0x55aa213d6557]
 8: (librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher(int)+0x275) [0x55aa213d64c5]
 9: (librbd::util::detail::C_CallbackAdapter<librbd::image::CloseRequest<librbd::ImageCtx>, &librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher>::finish(int)+0x28) [0x55aa213d753e]
 10: (Context::complete(int)+0x2b) [0x55aa2099c367]
 11: (librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}::operator()() const+0x38) [0x55aa2099c420]
 12: (void boost::asio::asio_handler_invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, ...)+0x6c) [0x55aa209ca904]
 13: (void boost_asio_handler_invoke_helpers::invoke<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, {lambda()#1}>(librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}&, {lambda()#1}&)+0x37) [0x55aa209ca01c]
 14: (void boost::asio::detail::handler_work<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>, void>::complete<{lambda()#1}>({lambda()#1}&, {lambda()#1}&)+0x3e) [0x55aa209c82e4]
 15: (boost::asio::detail::completion_handler<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xbf) [0x55aa209c588d]
 16: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x3a) [0x55aa211f55aa]
 17: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xa6) [0x55aa211f5d76]
 18: (boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long)+0x3a) [0x55aa211f55aa]
 19: (boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)+0x1c6) [0x55aa215bbac8]
 20: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0xfe) [0x55aa215bb672]
 21: (boost::asio::io_context::run()+0x4c) [0x55aa219b7c7c]
 22: (ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const+0x23) [0x55aa219b832d]
 23: (void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x24) [0x55aa219bdabc]
 24: (std::__invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x24) [0x55aa219bce90]
 25: (std::invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...)+0x24) [0x55aa219bbe14]
 26: _ZZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_ENKUlSA_SD_E_clIS3_JEEEDaSA_SD_()
 27: (std::basic_string_view<char, std::char_traits<char> > std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&, {lambda()#1}&&)+0x3b) [0x55aa219c03ce]
 28: (std::__invoke_result<std::basic_string_view<char, std::char_traits<char> >, (make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)...>::type std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}>(make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}&&)+0x3b) [0x55aa219c00f5]
 29: (void std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)+0x47) [0x55aa219bfea5]
 30: (std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> >::operator()()+0x1c) [0x55aa219bfb98]
 31: (std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1, auto:2&&)#1}, {lambda()#1}> > >::_M_run()+0x20) [0x55aa219bf4b0]

Actions

Copy link

Updated by Kefu Chai almost 3 years ago

[ RUN      ] TestImageReplayer/3.SnapshotProtect
flushed
remote_snap_id=6
local_snap_id=5, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1018ef94-55f6-4e08-a959-7da790271df8, primary_snap_id=6, last_copied_object_number=0, snap_seqs={5=4,6=18446744073709551614}]
flushed
remote_snap_id=7
local_snap_id=6, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1018ef94-55f6-4e08-a959-7da790271df8, primary_snap_id=7, last_copied_object_number=0, snap_seqs={5=4,7=18446744073709551614}]
../src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f5404ff9700 time 2021-07-07T17:36:39.262889+0000
../src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)
 ceph version Development (no_version) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a1) [0x7f54c1beae8f]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x16ad12d) [0x7f54c1beb12d]
 3: (AsyncOpTracker::~AsyncOpTracker()+0x49) [0x7f54c1acd69d]
 4: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x6c) [0x55faffeca856]
 5: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x1c) [0x55faffeca8ce]
 6: (librbd::ImageCtx::shutdown()+0x2f) [0x55faffe99409]
 7: (librbd::image::CloseRequest<librbd::ImageCtx>::finish()+0x1f) [0x55fb0006b2e5]
 8: (librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher(int)+0x275) [0x55fb0006b253]

Actions

Copy link

Updated by Deepika Upadhyay almost 3 years ago

@Kefu Chai is this from local or jenkins?

Actions

Copy link

Updated by Neha Ojha over 2 years ago

[ RUN      ] TestImageReplayer/3.SnapshotProtect
flushed
remote_snap_id=6
local_snap_id=5, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f65ea920-cc96-47cb-aaac-2e3ab57d1479, primary_snap_id=6, last_copied_object_number=0, snap_seqs={5=4,6=18446744073709551614}]
flushed
remote_snap_id=7
local_snap_id=6, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f65ea920-cc96-47cb-aaac-2e3ab57d1479, primary_snap_id=7, last_copied_object_number=0, snap_seqs={5=4,7=18446744073709551614}]
../src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7fd06bff7700 time 2021-12-01T19:59:22.678701+0000
../src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)
 ceph version Development (no_version) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1db) [0x7fd1274722eb]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x140210f) [0x7fd12747210f]
 3: (AsyncOpTracker::~AsyncOpTracker()+0x49) [0x7fd127344189]
 4: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x72) [0x2ec07c2]
 5: (librbd::ImageWatcher<librbd::ImageCtx>::~ImageWatcher()+0x1c) [0x2ec086c]
 6: (librbd::ImageCtx::shutdown()+0x36) [0x2e92d66]
 7: (librbd::image::CloseRequest<librbd::ImageCtx>::finish()+0x1c) [0x3083d1c]
 8: (librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher(int)+0x35c) [0x308410c]
 9: (librbd::util::detail::C_CallbackAdapter<librbd::image::CloseRequest<librbd::ImageCtx>, &librbd::image::CloseRequest<librbd::ImageCtx>::handle_flush_image_watcher>::finish(int)+0x68) [0x3084cf8]
 10: (Context::complete(int)+0x22) [0x25a22b2]

https://jenkins.ceph.com/job/ceph-pull-requests/86586/consoleFull#12738589407725d52c-2930-43cb-b77a-ae0a919c2170

Actions

Copy link