Project

General

Profile

Actions

Bug #49568

closed

net: error in make check

Added by Xuehan Xu about 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

INFO 2021-03-02 04:53:25,110 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4@52028 >> osd.2 v2:127.0.0.1:9012/2] connected: gs=3, pgs=3, cs=0, client_cookie=17764031311294571930, server_cookie=14191085562580658863, in_seq=0, out_seq=0, out_q=0
unittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed. =================================================================
10520ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004b90 at pc 0x55dcfac378b8 bp 0x7ffd757bf4e0 sp 0x7ffd757bf4d0
READ of size 8 at 0x619000004b90 thread T0
#0 0x55dcfac378b7 in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x55dcfac3fd77 in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x55dcfac3fa51 in _find_if<_gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, _gnu_cxx::_ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x55dcfac3f455 in _find_if<_gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, _gnu_cxx::_ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x55dcfac3e7f7 in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x55dcfac37b05 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
#6 0x55dcfa6073f3 in void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/util/backtrace.hh:62
#7 0x55dcfa5df5d5 in seastar::backtrace_buffer::append_backtrace() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:732
#8 0x55dcfa48ff10 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:753
#9 0x55dcfa490083 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:760
#10 0x55dcfa4f3e23 in sigabrt_action /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3513
#11 0x55dcfa52c9ee in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3495
#12 0x55dcfa52ca89 in _FUN /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3491
#13 0x7f1762cd389f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#14 0x7f176141af46 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x3ef46)
#15 0x7f176141c8b0 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b0)
#16 0x7f176140c429 (/lib/x86_64-linux-gnu/libc.so.6+0x30429)
#17 0x7f176140c4a1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x304a1)
#18 0x55dcf9da2f91 in seastar::sharded<crimson::common::ConfigProxy>::~sharded() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518
#19 0x7f176141f0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
#20 0x7f176141f1e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
#21 0x7f17613fdb9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
#22 0x55dcf7bde869 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger+0xb1aa869)

0x619000004b90 is located 16 bytes inside of 1024-byte region [0x619000004b80,0x619000004f80)
freed by thread T0 here:
#0 0x7f1764753cef in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ecef)
#1 0x55dcfac4d083 in __gnu_cxx::new_allocator<seastar::shared_object>::deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x55dcfac499d3 in std::allocator_traits<std::allocator<seastar::shared_object> >::deallocate(std::allocator<seastar::shared_object>&, seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x55dcfac44eb9 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x55dcfac427dc in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x55dcfac429f9 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7f176141f0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)

previously allocated by thread T0 here:
#0 0x7f1764752d6f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dd6f)
#1 0x55dcfac4fe0e in _gnu_cxx::new_allocator<seastar::shared_object>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x55dcfac4e859 in std::allocator_traits<std::allocator<seastar::shared_object> >::allocate(std::allocator<seastar::shared_object>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x55dcfac4cb9f in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x55dcfac490c0 in void std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_realloc_insert<seastar::shared_object>(
_gnu_cxx::__normal_iterator<seastar::shared_object*, std::vector<seastar::shared_object, std::allocator<seastar::shared_object> > >, seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:440
#5 0x55dcfac44c5b in seastar::shared_object& std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::emplace_back<seastar::shared_object>(seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:121
#6 0x55dcfac425b0 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::push_back(seastar::shared_object&&) /usr/include/c++/9/bits/stl_vector.h:1201
#7 0x55dcfac374b9 in dl_iterate_phdr_callback /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:49
#8 0x7f1761542160 in dl_iterate_phdr (/lib/x86_64-linux-gnu/libc.so.6+0x166160)

SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76 in operator()
Shadow bytes around the buggy address:
0x0c327fff8920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8950: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff8960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff8970: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8990: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
10520ABORTING

Actions #1

Updated by Yingxin Cheng about 3 years ago

see https://tracker.ceph.com/issues/48672

I guess there was another error log saying "test_messenger timeout after 120s, abort! Consider to extend the period if the test is still running."

Which means it might be worthwhile to extend the 120s limit if the issue happens too frequently.

Actions #2

Updated by Yingxin Cheng almost 2 years ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF