Project

General

Profile

Bug #40368

[rbd-mirror] image sync can crash when updating progress

Added by Jason Dillaman almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The invocation of "m_copied_objects.top()" is not guarded so it might be invoked on an empty heap, resulting in an assertion failure.

#0  0x00007fffeb74c93f in raise () from /lib64/libc.so.6
#1  0x00007fffeb736c95 in abort () from /lib64/libc.so.6
#2  0x0000555555770d08 in std::__replacement_assert (__file=__file@entry=0x555555c1af78 "/usr/include/c++/8/bits/stl_queue.h", __line=__line@entry=582, 
    __function=__function@entry=0x555555c1b0c0 <std::priority_queue<unsigned long, std::vector<unsigned long, std::allocator<unsigned long> >, std::greater<unsigned long> >::top() const::__PRETTY_FUNCTION__> "std::priority_queue<_Tp, _Sequence, _Compare>::const_reference std::priority_queue<_Tp, _Sequence, _Compare>::top() const [with _Tp = long unsigned int; _Sequence = std::vector<long unsigned int>; _Co"..., 
    __condition=__condition@entry=0x555555bd61e8 "__builtin_expect(!this->empty(), true)") at /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h:2391
#3  0x000055555599f3bf in std::priority_queue<unsigned long, std::vector<unsigned long, std::allocator<unsigned long> >, std::greater<unsigned long> >::top (this=0x7fff440b0318)
    at /usr/include/c++/8/bits/stl_iterator.h:783
#4  librbd::deep_copy::ImageCopyRequest<librbd::ImageCtx>::handle_object_copy (this=0x7fff440b0200, object_no=<optimized out>, r=<optimized out>)
    at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/librbd/deep_copy/ImageCopyRequest.cc:146
#5  0x0000555555777730 in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>)
    at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/build/boost/include/boost/function/function_template.hpp:682
#6  FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/include/Context.h:487
#7  0x0000555555775b0d in Context::complete (this=0x7fff740fbb40, r=<optimized out>) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/include/Context.h:77
#8  0x00005555559ac985 in librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::finish (this=this@entry=0x7fff740f37e0, r=r@entry=-2) at /usr/include/c++/8/ext/new_allocator.h:86
#9  0x00005555559af345 in librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::handle_read_from_parent (this=this@entry=0x7fff740f37e0, r=r@entry=0) at /usr/include/c++/8/bits/stl_map.h:463
#10 0x00005555559afdf7 in librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::send_read_from_parent (this=0x7fff740f37e0) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/common/RWLock.h:123
#11 0x00005555559b05ac in librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::send_read_object (this=this@entry=0x7fff740f37e0) at /usr/include/c++/8/bits/stl_tree.h:352
#12 0x00005555559b0eb0 in librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::handle_list_snaps (this=0x7fff740f37e0, r=<optimized out>)
    at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/log/Entry.h:35
#13 0x00007ffff7b12851 in librados::C_AioComplete::finish (this=0x7fff741080f0, r=<optimized out>) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/librados/AioCompletionImpl.h:167
#14 0x00007ffff7ad762d in Context::complete (this=0x7fff741080f0, r=<optimized out>) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/include/Context.h:77
#15 0x00007fffeef13f75 in Finisher::finisher_thread_entry (this=0x5555562b4750) at /usr/src/debug/ceph-14.2.1-385.g4ae8136.el8cp.x86_64/src/common/Finisher.cc:67
#16 0x00007fffed2a22de in start_thread () from /lib64/libpthread.so.0
#17 0x00007fffeb811a63 in clone () from /lib64/libc.so.6

Related issues

Copied to rbd - Backport #40379: nautilus: [rbd-mirror] image sync can crash when updating progress Resolved
Copied to rbd - Backport #40380: mimic: [rbd-mirror] image sync can crash when updating progress Resolved

History

#1 Updated by Jason Dillaman almost 5 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 28559

#2 Updated by Jason Dillaman almost 5 years ago

  • Subject changed from [rbd-mirror] image sync can crash due to out-of-order object copying to [rbd-mirror] image sync can crash when updating progress

#3 Updated by Mykola Golub almost 5 years ago

  • Status changed from Fix Under Review to Pending Backport

#4 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #40379: nautilus: [rbd-mirror] image sync can crash when updating progress added

#5 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #40380: mimic: [rbd-mirror] image sync can crash when updating progress added

#6 Updated by Nathan Cutler over 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF