Project

General

Profile

Bug #44311

crash in Objecter and CRUSH map lookup

Added by Mahati Chamarthy about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Jason Dillaman
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
CRUSH, Objecter, librados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When Concurrent reads are issued with the below rbd command, it results in failure due to crash in Objecter and CRUSH map lookup. (Full log report attached as txt file)

rbd export test@snap - --log-file test.log --debug-ms 1 --rbd-localize-snap-reads

==115801==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700003aa48 at pc 0x7fd0b657fc97 bp 0x7fd0a6a75f90 sp 0x7fd0a6a75f80
READ of size 8 at 0x60700003aa48 thread T20 (tp_librbd)
    #0 0x7fd0b657fc96 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::size() const /usr/include/c++/8/bits/basic_string.h
:931
    #1 0x7fd0b65a282b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_
traits<char>, std::allocator<char> > const&) const /usr/include/c++/8/bits/basic_string.h:2845
    #2 0x7fd0b659ea11 in bool std::operator< <char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, s
td::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/include/c++/8/bits/basic_string.h:
6136
    #3 0x7fd0b65aa95e in std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator()(std::__cxx11::basic_string<c
har, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /us
r/include/c++/8/bits/stl_function.h:386
    #4 0x7fd0b673438f in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<
char, std::char_traits<char>, std::allocator<char> > const, int>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::all
ocator<char> > const, int> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cx
x11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_strin
g<char, std::char_traits<char>, std::allocator<char> > const, int> >*, std::_Rb_tree_node_base*, std::__cxx11::basic_string<char, std::char_traits<char>, std
::allocator<char> > const&) /usr/include/c++/8/bits/stl_tree.h:1891
    #5 0x7fd0b672d3fa in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<
char, std::char_traits<char>, std::allocator<char> > const, int>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::all
ocator<char> > const, int> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cx
x11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) /usr/include/c++/8/bits/stl_tree.h:1206
    #6 0x7fd0b672415a in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::less<std::__cxx11::basic_string<
char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char
> > const, int> > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/include/c++/8/bits/stl_map.h:12
39
    #7 0x7fd0b671ae7e in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::less<std::__cxx11::basic_string<
char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char
> > const, int> > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/include/c++/8/bits/stl_map.h:495
    #8 0x7fd0b75852d2 in CrushWrapper::build_rmap(std::map<int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<in
t>, std::allocator<std::pair<int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::map<std::__cxx11::
basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<cha
r> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > >&) const /media/cephdir/ceph/
src/crush/CrushWrapper.h:90
    #9 0x7fd0b758506c in CrushWrapper::build_rmaps() const /media/cephdir/ceph/src/crush/CrushWrapper.h:82
    #10 0x7fd0b758620b in CrushWrapper::get_item_id(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /media/ceph
dir/ceph/src/crush/CrushWrapper.h:426
    #11 0x7fd0b7a0e31a in CrushWrapper::get_full_location_ordered(int, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::al
locator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char,
 std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) const /media/cephdir
/ceph/src/crush/CrushWrapper.cc:796
    #12 0x7fd0b7a0dcc7 in CrushWrapper::get_full_location[abi:cxx11](int) const /media/cephdir/ceph/src/crush/CrushWrapper.cc:764
    #13 0x7fd0b7a0b4fb in CrushWrapper::get_common_ancestor_distance(CephContext*, int, std::multimap<std::__cxx11::basic_string<char, std::char_traits<char>
, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, st
d::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) const /media/cephdir/ceph/src/crush/CrushWrapper.cc:651
    #14 0x7fd0c1467abd in Objecter::_calc_target(Objecter::op_target_t*, Connection*, bool) /media/cephdir/ceph/src/osdc/Objecter.cc:2971
    #15 0x7fd0c145d669 in Objecter::_op_submit(Objecter::Op*, ceph::shunique_lock<std::shared_mutex>&, unsigned long*) /media/cephdir/ceph/src/osdc/Objecter.
cc:2379
    #16 0x7fd0c145c40e in Objecter::_op_submit_with_budget(Objecter::Op*, ceph::shunique_lock<std::shared_mutex>&, unsigned long*, int*) /media/cephdir/ceph/
src/osdc/Objecter.cc:2296
    #17 0x7fd0c145bd0f in Objecter::op_submit(Objecter::Op*, unsigned long*, int*) /media/cephdir/ceph/src/osdc/Objecter.cc:2263
    #18 0x7fd0c13d31b0 in librados::IoCtxImpl::aio_operate_read(object_t const&, ObjectOperation*, librados::AioCompletionImpl*, int, ceph::buffer::v14_2_0::
list*, blkin_trace_info const*) /media/cephdir/ceph/src/librados/IoCtxImpl.cc:747
    #19 0x7fd0c1355596 in librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, libra
dos::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectReadOperation*, int, ceph::buffer::v14_2_0::list*, blkin_trace_info const*) /media/cephdir/ceph/src/li
brados/librados_cxx.cc:1622
    #20 0x7fd0c2643a75 in librbd::io::ObjectReadRequest<librbd::ImageCtx>::read_object() /media/cephdir/ceph/src/librbd/io/ObjectRequest.cc:233
    #21 0x7fd0c2642ef7 in librbd::io::ObjectReadRequest<librbd::ImageCtx>::send() /media/cephdir/ceph/src/librbd/io/ObjectRequest.cc:195
    #22 0x7fd0c263d0f7 in librbd::io::ObjectDispatch<librbd::ImageCtx>::read(unsigned long, unsigned long, unsigned long, unsigned long, int, ZTracer::Trace
const&, ceph::buffer::v14_2_0::list*, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned
 long> > >*, int*, librbd::io::DispatchResult*, Context**, Context*) /media/cephdir/ceph/src/librbd/io/ObjectDispatch.cc:50
    #23 0x7fd0c23bd4a6 in librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor::operator()(librbd::io::ObjectDispatchSpec::ReadRequest&) const /media/
cephdir/ceph/src/librbd/io/ObjectDispatcher.cc:109
    #24 0x7fd0c23c4f19 in boost::disable_if_c<(false)&&boost::is_same<librbd::io::ObjectDispatchSpec::ReadRequest&, librbd::io::ObjectDispatchSpec::ReadReque
st&>::value, bool>::type boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::internal_visit<lib
rbd::io::ObjectDispatchSpec::ReadRequest&>(librbd::io::ObjectDispatchSpec::ReadRequest&, int) /media/cephdir/ceph/build/boost/include/boost/variant/variant.h
pp:1058
    #25 0x7fd0c23c4b72 in boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::result_type boost
::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, fals
e>, void*, librbd::io::ObjectDispatchSpec::ReadRequest>(int, boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisit
or const, false>&, void*, librbd::io::ObjectDispatchSpec::ReadRequest*, mpl_::bool_<true>) /media/cephdir/ceph/build/boost/include/boost/variant/detail/visit
ation_impl.hpp:114
    #26 0x7fd0c23c4348 in boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::result_type boost
::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>, v
oid*, librbd::io::ObjectDispatchSpec::ReadRequest, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest
, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, lib
rbd::io::ObjectDispatchSpec::FlushRequest>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::S
endVisitor const, false>&, void*, librbd::io::ObjectDispatchSpec::ReadRequest*, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::Objec
tDispatchSpec::DiscardRequest, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec
::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec::FlushRequest>::has_fallback_type_, int) /media/cephdir/ceph/build/boost/include/boost/variant/detai
l/visitation_impl.hpp:154
    #27 0x7fd0c23c3ad7 in boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::result_type boost
::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<6l>, librbd:
:io::ObjectDispatchSpec::ReadRequest, boost::mpl::l_item<mpl_::long_<5l>, librbd::io::ObjectDispatchSpec::DiscardRequest, boost::mpl::l_item<mpl_::long_<4l>,
 librbd::io::ObjectDispatchSpec::WriteRequest, boost::mpl::l_item<mpl_::long_<3l>, librbd::io::ObjectDispatchSpec::WriteSameRequest, boost::mpl::l_item<mpl_:
:long_<2l>, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, boost::mpl::l_item<mpl_::long_<1l>, librbd::io::ObjectDispatchSpec::FlushRequest, boost::mpl::l_end> > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::Send
Visitor const, false>, void*, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectD
ispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::ObjectDispat
chSpec::FlushRequest>::has_fallback_type_>(int, int, boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const
, false>&, void*, mpl_::bool_<false>, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io:
:ObjectDispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::Obje
ctDispatchSpec::FlushRequest>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::l
ong_<6l>, librbd::io::ObjectDispatchSpec::ReadRequest, boost::mpl::l_item<mpl_::long_<5l>, librbd::io::ObjectDispatchSpec::DiscardRequest, boost::mpl::l_item
<mpl_::long_<4l>, librbd::io::ObjectDispatchSpec::WriteRequest, boost::mpl::l_item<mpl_::long_<3l>, librbd::io::ObjectDispatchSpec::WriteSameRequest, boost::
mpl::l_item<mpl_::long_<2l>, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, boost::mpl::l_item<mpl_::long_<1l>, librbd::io::ObjectDispatchSpec::Flus
hRequest, boost::mpl::l_end> > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*) /media/cephdir/ceph/build/boost/include/boost/variant/detail/visitation_i
mpl.hpp:238
    #28 0x7fd0c23c370f in boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::result_type boost
::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::
io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec::FlushRequest>::internal_app
ly_visitor_impl<boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>, void*>(int, int, boost::det
ail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>&, void*) /media/cephdir/ceph/build/boost/include/boost
/variant/variant.hpp:2452
    #29 0x7fd0c23c2fea in boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>::result_type boost
::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::
io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec::FlushRequest>::internal_app
ly_visitor<boost::detail::variant::invoke_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false> >(boost::detail::variant::invoke_
visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const, false>&) /media/cephdir/ceph/build/boost/include/boost/variant/variant.hpp:2466
    #30 0x7fd0c23c0b5d in librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const::result_type boost::variant<librbd::io::ObjectDispatchSpec::ReadR
equest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librb
d::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec::FlushRequest>::apply_visitor<librbd::io::ObjectDispatcher<librbd::ImageCtx
>::SendVisitor const>(librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const&) & /media/cephdir/ceph/build/boost/include/boost/variant/variant.hpp
:2518
    #31 0x7fd0c23bf5e6 in librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor::result_type boost::apply_visitor<librbd::io::ObjectDispatcher<librbd::
ImageCtx>::SendVisitor, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io::ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectDispatc
hSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispatchSpec::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec
::FlushRequest>&>(librbd::io::ObjectDispatcher<librbd::ImageCtx>::SendVisitor const&, boost::variant<librbd::io::ObjectDispatchSpec::ReadRequest, librbd::io:
:ObjectDispatchSpec::DiscardRequest, librbd::io::ObjectDispatchSpec::WriteRequest, librbd::io::ObjectDispatchSpec::WriteSameRequest, librbd::io::ObjectDispat
chSpec::CompareAndWriteRequest, librbd::io::ObjectDispatchSpec::FlushRequest>&) /media/cephdir/ceph/build/boost/include/boost/variant/detail/apply_visitor_un
ary.hpp:97
    #32 0x7fd0c23bca2a in librbd::io::ObjectDispatcher<librbd::ImageCtx>::send(librbd::io::ObjectDispatchSpec*) /media/cephdir/ceph/src/librbd/io/ObjectDispa
tcher.cc:336
    #33 0x7fd0c23b84c0 in librbd::io::ObjectDispatchSpec::send() /media/cephdir/ceph/src/librbd/io/ObjectDispatchSpec.cc:37
    #34 0x7fd0c237c2d8 in librbd::io::ImageReadRequest<librbd::ImageCtx>::send_request() (/usr/local/lib/librbd.so.1+0xa522d8)
    #35 0x7fd0c2379d55 in librbd::io::ImageRequest<librbd::ImageCtx>::send() (/usr/local/lib/librbd.so.1+0xa4fd55)
    #36 0x7fd0c2378d2c in librbd::io::ImageRequest<librbd::ImageCtx>::aio_read(librbd::ImageCtx*, librbd::io::AioCompletion*, std::vector<std::pair<unsigned
long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long> > >&&, librbd::io::ReadResult&&, int, ZTracer::Trace const&) (/usr/local/lib/lib
rbd.so.1+0xa4ed2c)
    #37 0x7fd0c236be2e in librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor::operator()(librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read&) const /media/cephdir/ceph/src/librbd/io/ImageDispatchSpec.cc:23
    #38 0x7fd0c2373de9 in boost::disable_if_c<(false)&&boost::is_same<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read&, librbd::io::ImageDispatchSpec<l
ibrbd::ImageCtx>::Read&>::value, void>::type boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false
>::internal_visit<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read&>(librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read&, int) /media/cephdir/ceph/bu
ild/boost/include/boost/variant/variant.hpp:1058
    #39 0x7fd0c237322c in boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>::result_type boos
t::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, fa
lse>, void*, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read>(int, boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCt
x>::SendVisitor const, false>&, void*, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read*, mpl_::bool_<true>) /media/cephdir/ceph/build/boost/include/boo
st/variant/detail/visitation_impl.hpp:114
    #40 0x7fd0c2371a7d in boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>::result_type boos
t::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>,
 void*, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, boost::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatc
hSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd:
:io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::has_fallback_type_>(int, boost::detail::v
ariant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>&, void*, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::R
ead*, boost::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispat
chSpec<librbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite
, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::has_fallback_type_, int) /media/cephdir/ceph/build/boost/include/boost/variant/detail/visitation_i
mpl.hpp:154
    #41 0x7fd0c236f8b6 in boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>::result_type boos
t::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<6l>, librbd
::io::ImageDispatchSpec<librbd::ImageCtx>::Read, boost::mpl::l_item<mpl_::long_<5l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, boost::mpl::l_
item<mpl_::long_<4l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, boost::mpl::l_item<mpl_::long_<3l>, librbd::io::ImageDispatchSpec<librbd::Image
Ctx>::WriteSame, boost::mpl::l_item<mpl_::long_<2l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, boost::mpl::l_item<mpl_::long_<1l>, li
brbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush, boost::mpl::l_end> > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke
_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>, void*, boost::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::R
ead, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::
ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::has_fallback
_type_>(int, int, boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>&, void*, mpl_::bool_<fals
e>, boost::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatch
Spec<librbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite,
librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<
boost::mpl::l_item<mpl_::long_<6l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, boost::mpl::l_item<mpl_::long_<5l>, librbd::io::ImageDispatchSpec<
librbd::ImageCtx>::Discard, boost::mpl::l_item<mpl_::long_<4l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, boost::mpl::l_item<mpl_::long_<3l>, l
ibrbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, boost::mpl::l_item<mpl_::long_<2l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWri
te, boost::mpl::l_item<mpl_::long_<1l>, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush, boost::mpl::l_end> > > > > > >, boost::mpl::l_iter<boost::mpl
::l_end> >*) /media/cephdir/ceph/build/boost/include/boost/variant/detail/visitation_impl.hpp:238
    #42 0x7fd0c236e3f0 in boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>::result_type boos
t::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<lib
rbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::
io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::internal_apply_visitor_impl<boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::Ima
geCtx>::SendVisitor const, false>, void*>(int, int, boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>&, void*) /media/cephdir/ceph/build/boost/include/boost/variant/variant.hpp:2452
    #43 0x7fd0c236dce8 in boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>::result_type boos
t::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<lib
rbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::
io::ImageDispatchSpec<librbd::ImageCtx>::Flush>::internal_apply_visitor<boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx
>::SendVisitor const, false> >(boost::detail::variant::invoke_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const, false>&) /media/cep
hdir/ceph/build/boost/include/boost/variant/variant.hpp:2466
    #44 0x7fd0c236d367 in librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const::result_type boost::variant<librbd::io::ImageDispatchSpec<librbd
::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, librbd::io::ImageDispatc
hSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>
::apply_visitor<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const>(librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const&) & /m
edia/cephdir/ceph/build/boost/include/boost/variant/variant.hpp:2518
    #45 0x7fd0c236cb52 in librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor::result_type boost::apply_visitor<librbd::io::ImageDispatchSpec<librbd
::ImageCtx>::SendVisitor, boost::variant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, lib
rbd::io::ImageDispatchSpec<librbd::ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageC
tx>::CompareAndWrite, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Flush>&>(librbd::io::ImageDispatchSpec<librbd::ImageCtx>::SendVisitor const&, boost::v
ariant<librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Read, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::Discard, librbd::io::ImageDispatchSpec<librbd:
:ImageCtx>::Write, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::WriteSame, librbd::io::ImageDispatchSpec<librbd::ImageCtx>::CompareAndWrite, librbd::io::
ImageDispatchSpec<librbd::ImageCtx>::Flush>&) /media/cephdir/ceph/build/boost/include/boost/variant/detail/apply_visitor_unary.hpp:97
    #46 0x7fd0c236b02b in librbd::io::ImageDispatchSpec<librbd::ImageCtx>::send() /media/cephdir/ceph/src/librbd/io/ImageDispatchSpec.cc:115
    #47 0x7fd0c23a761f in librbd::io::ImageRequestWQ<librbd::ImageCtx>::process_io(librbd::io::ImageDispatchSpec<librbd::ImageCtx>*, bool) (/usr/local/lib/li
brbd.so.1+0xa7d61f)
    #48 0x7fd0c23a2f6e in librbd::io::ImageRequestWQ<librbd::ImageCtx>::process(librbd::io::ImageDispatchSpec<librbd::ImageCtx>*) (/usr/local/lib/librbd.so.1
+0xa78f6e)
    #49 0x7fd0c23b7b36 in ThreadPool::PointerWQ<librbd::io::ImageDispatchSpec<librbd::ImageCtx> >::_void_process(void*, ThreadPool::TPHandle&) /media/cephdir
/ceph/src/common/WorkQueue.h:412
    #50 0x7fd0b684cf89 in ThreadPool::worker(ThreadPool::WorkThread*) /media/cephdir/ceph/src/common/WorkQueue.cc:118
    #51 0x7fd0b6856020 in ThreadPool::WorkThread::entry() /media/cephdir/ceph/src/common/WorkQueue.h:466
    #52 0x7fd0b67ff3cc in Thread::entry_wrapper() /media/cephdir/ceph/src/common/Thread.cc:84
    #53 0x7fd0b67ff243 in Thread::_entry_func(void*) /media/cephdir/ceph/src/common/Thread.cc:71
    #54 0x7fd0b4b536b9 in start_thread /build/glibc-LK5gWL/glibc-2.23/nptl/pthread_create.c:333
    #55 0x7fd0b393c41c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

crash_report.txt View (69.2 KB) Mahati Chamarthy, 02/26/2020 05:45 PM


Related issues

Copied to RADOS - Backport #46595: octopus: crash in Objecter and CRUSH map lookup Resolved
Copied to RADOS - Backport #49073: nautilus: crash in Objecter and CRUSH map lookup Resolved

History

#1 Updated by Neha Ojha about 4 years ago

  • Status changed from New to Need More Info

Which version is this on?

#2 Updated by Mahati Chamarthy about 4 years ago

Neha Ojha wrote:

Which version is this on?

Current master

#3 Updated by Mahati Chamarthy about 4 years ago

Mahati Chamarthy wrote:

Neha Ojha wrote:

Which version is this on?

Current master. To reproduce set rbd_op_thread = 8 in ceph.conf and run the command given in this report.

#4 Updated by Mahati Chamarthy about 4 years ago

To give more context, this issue is blocking progress on rbd op threads config change -> https://github.com/ceph/ceph/pull/24483

#5 Updated by Neha Ojha about 4 years ago

Is this something that started appearing recently? Do you have a commit or version that works for this same command? Might be useful to know that to narrow down the culprit. Also, this may be related to https://tracker.ceph.com/issues/44373.

#6 Updated by Mahati Chamarthy about 4 years ago

Neha Ojha wrote:

Is this something that started appearing recently? Do you have a commit or version that works for this same command?

It's been there always I suppose. This got attention now, as all the other race conditions seem to be resolved. By setting 'rbd_op_threads = 8' in ceph.conf and creating an rbd image and a snapshot:

rbd create --image-feature layering -s 10 test
rbd snap create test@snap

running this below command in a loop should reproduce the issue:

rbd export test@snap - --log-file test.log --debug-ms 1 --rbd-localize-snap-reads

#7 Updated by Mahati Chamarthy about 4 years ago

Scratch that. If you replace qa/workunits/rbd/read-flags.sh with this script https://gist.github.com/MahatiC/a4bf43100740df9377001f2bb3718146 and set 'rbd_op_threads = 8' in ceph.conf. And then if you execute the script in a loop, it should be reproducible.

#8 Updated by Neha Ojha about 4 years ago

  • Status changed from Need More Info to New
  • Priority changed from Normal to High

#9 Updated by Neha Ojha about 4 years ago

Conversation from IRC:

<neha> mahatic: I would like to know Aemerson's thoughts on https://tracker.ceph.com/issues/44311 though
<Aemerson> Well if I'm reading this right it seems we could try to guard build-rmaps with a lock analogous to what's done for static function-level variables so only one thread can get into the builder.
<Aemerson> Or build them in create.
<Aemerson> THOUGH
<Aemerson> We modify the rmaps at a few places. So maybe we just want to put a shared_mutex over it so we don't have it either change out from under someone OR have two people rush in to do the same modification.

#10 Updated by Jason Dillaman over 3 years ago

  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman

#11 Updated by Jason Dillaman over 3 years ago

I am hitting this all the time now that librbd is using the 'neorados' API [1]. I plan to just rebuild the rmaps when decoding the crush map since it's invalidated at this point. The other methods that cause invalidation don't seem to be invoked by Objecter code paths.

[1] http://qa-proxy.ceph.com/teuthology/jdillaman-2020-07-13_18:23:39-rbd-wip-jd-testing-distro-basic-smithi/5223828/teuthology.log

#12 Updated by Jason Dillaman over 3 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 36103

#13 Updated by Jason Dillaman over 3 years ago

  • Backport set to octopus

#14 Updated by Kefu Chai over 3 years ago

  • Status changed from Fix Under Review to Pending Backport

#15 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #46595: octopus: crash in Objecter and CRUSH map lookup added

#16 Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

#17 Updated by Kefu Chai about 3 years ago

  • Copied to Backport #49073: nautilus: crash in Objecter and CRUSH map lookup added

Also available in: Atom PDF