Bug #44391: unittest_rbd_mirror fails in "TestImageReplayerJournal.ErrorNoJournal" - rbd - Ceph

Actions

Copy link

Bug #44391

closed

unittest_rbd_mirror fails in "TestImageReplayerJournal.ErrorNoJournal"

Added by Kefu Chai about 4 years ago. Updated about 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

33988

Crash signature (v1):

Crash signature (v2):

Description


181: [ RUN      ] TestImageReplayerJournal.ErrorNoJournal
181: terminate called after throwing an instance of 'std::length_error'
181:   what():  basic_string::_M_create
1/1 Test #181: unittest_rbd_mirror ..............***Exception: Child aborted 63.21 sec

in a91c3112a9364a17f846d80ceadedb187ef0510f, and in a bonic docker container env:

i was running

while ctest -R unittest_rbd_mirror  -V; do sleep 1; done

in hope to reproduce the random failure we have in jenkins "make check" runs.

Jason, mind taking a look?

Actions

Copy link

Updated by Kefu Chai about 4 years ago

Description updated (diff)

Actions

Copy link

Updated by Jason Dillaman about 4 years ago

I had these tests running all day yesterday in a loop and I never caught that failure, unfortunately.

Actions

Copy link

Updated by Jason Dillaman about 4 years ago

Status changed from New to Need More Info
Assignee deleted (~~Jason Dillaman~~)

Actions

Copy link

Updated by Kefu Chai about 4 years ago

[ RUN      ] TestImageReplayerJournal.MirroringDelay
get_commit_positions: client: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
get_commit_positions: client: [id=5a0f181a-5bab-4d5b-8c8c-8ccc391e8505, commit_position=[positions=[]], state=connected]
get_commit_positions: client: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
get_commit_positions: client: [id=5a0f181a-5bab-4d5b-8c8c-8ccc391e8505, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f9293e8b801 in __GI_abort () at abort.c:79
#2  0x00007f929488b1f2 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f9294896e36 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f9294896e81 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f92948970b5 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f929488ae91 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000558354ebc026 in __gnu_cxx::new_allocator<char>::allocate (this=0x7f9243ffa7d0, __n=46242909976937) at /usr/include/c++/9/ext/new_allocator.h:114
#8  0x0000558354ebbe8a in std::allocator_traits<std::allocator<char> >::allocate (__a=..., __n=46242909976937) at /usr/include/c++/9/bits/alloc_traits.h:444
#9  0x0000558354ebbd1f in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create (this=0x7f9243ffa7d0, __capacity=@0x7f9243ffa690: 46242909976936, __old_capacity=0) at /usr/include/c++/9/bits/basic_string.tcc:153
#10 0x0000558354ef5570 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (this=0x7f9243ffa7d0, __beg=0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb", __end=0x7f921c026ae8 "$")
    at /usr/include/c++/9/bits/basic_string.tcc:219
#11 0x0000558354eeb090 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char*> (this=0x7f9243ffa7d0, __beg=0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb", __end=0x7f921c026ae8 "$")
    at /usr/include/c++/9/bits/basic_string.h:247
#12 0x0000558354edebad in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (this=0x7f9243ffa7d0, __beg=0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb", __end=0x7f921c026ae8 "$")
    at /usr/include/c++/9/bits/basic_string.h:266
#13 0x0000558354ecabc2 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f9243ffa7d0, __str="d25176c6-dd9f-4a06-8885-e472570c89fb") at /usr/include/c++/9/bits/basic_string.h:451
#14 0x00005583552f9f88 in rbd::mirror::image_replayer::BootstrapRequest<librbd::ImageCtx>::get_local_image_name[abi:cxx11]() const (this=0x7f921c026960) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/tools/rbd_mirror/image_replayer/BootstrapRequest.cc:108
#15 0x0000558354ff29f1 in rbd::mirror::ImageReplayer<librbd::ImageCtx>::reregister_admin_socket_hook (this=0x7f921c0033b0) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/tools/rbd_mirror/ImageReplayer.cc:1041
#16 0x0000558354fee38b in rbd::mirror::ImageReplayer<librbd::ImageCtx>::set_mirror_image_status_update (this=0x7f921c0033b0, force=false, opt_state=...) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/tools/rbd_mirror/ImageReplayer.cc:676
#17 0x0000558354ffbb20 in rbd::mirror::ImageReplayer<librbd::ImageCtx>::update_mirror_image_status(bool, boost::optional<rbd::mirror::ImageReplayer<librbd::ImageCtx>::State> const&)::{lambda(int)#3}::operator()(int) const (this=0x7f921c0033b0, r=0)
    at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/tools/rbd_mirror/ImageReplayer.cc:665
#18 0x0000558355024fc0 in LambdaContext<rbd::mirror::ImageReplayer<librbd::ImageCtx>::update_mirror_image_status(bool, boost::optional<rbd::mirror::ImageReplayer<librbd::ImageCtx>::State> const&)::{lambda(int)#3}>::finish(int) (this=0x55835711b510, r=0)
    at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/include/Context.h:129
#19 0x0000558354f7dc07 in Context::complete (this=0x55835711b510, r=0) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/include/Context.h:77
#20 0x0000558355693629 in ContextWQ::process (this=0x7f9268048640, ctx=0x55835711b510) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/WorkQueue.h:627
#21 0x0000558355695242 in ThreadPool::PointerWQ<Context>::_void_process (this=0x7f9268048640, item=0x55835711b510, handle=...) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/WorkQueue.h:412
#22 0x00007f92965653b9 in ThreadPool::worker (this=0x55835709e5d0, wt=0x558357c3f890) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/WorkQueue.cc:118
#23 0x00007f9296569348 in ThreadPool::WorkThread::entry (this=0x558357c3f890) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/WorkQueue.h:466
#24 0x00007f92965428e8 in Thread::entry_wrapper (this=0x558357c3f890) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/Thread.cc:84
#25 0x00007f9296542866 in Thread::_entry_func (arg=0x558357c3f890) at /var/ceph/ceph-ubuntu-18.04-kefu/ceph/src/common/Thread.cc:71
#26 0x00007f929ff0a6db in start_thread (arg=0x7f9243fff700) at pthread_create.c:463
#27 0x00007f9293f6c88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) f 13
(gdb) p __str._M_dataplus
$22 = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb"}
(gdb) p __str._M_string_length
$23 = 36

(gdb) down
#12 0x0000558354edebad in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (this=0x7f9243ffa7d0, __beg=0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb", __end=0x7f921c026ae8 "$")
    at /usr/include/c++/9/bits/basic_string.h:266
(gdb) p __beg
$24 = 0x558357c97180 "d25176c6-dd9f-4a06-8885-e472570c89fb" 
(gdb) p __end
$25 = 0x7f921c026ae8 "$" 
(gdb) p/x __end - __beg
$26 = 0x2a0ec438f968

   │261           template<typename _InIterator>                                                                                                                                                                                                                               │
   │262             void                                                                                                                                                                                                                                                       │
   │263             _M_construct(_InIterator __beg, _InIterator __end)                                                                                                                                                                                                         │
   │264             {                                                                                                                                                                                                                                                          │
   │265               typedef typename std::__is_integer<_InIterator>::__type _Integral;                                                                                                                                                                                       │
  >│266               _M_construct_aux(__beg, __end, _Integral());                                                                                                                                                                                                             │
   │267             }

Actions

Copy link