Actions
Bug #49426
closed/src/seastar/include/seastar/core/sharded.hh:seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
Added by Neha Ojha about 3 years ago. Updated about 3 years ago.
Status:
Duplicate
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
nittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
=================================================================
==9224==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004b90 at pc 0x556cac18ba18 bp 0x7fff071529a0 sp 0x7fff07152990
READ of size 8 at 0x619000004b90 thread T0
#0 0x556cac18ba17 in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x556cac193ed7 in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x556cac193bb1 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x556cac1935b5 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x556cac192957 in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x556cac18bc65 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
Updated by Josh Durgin about 3 years ago
- Priority changed from Normal to Urgent
INFO 2021-02-23 00:54:24,427 [shard 0] ms - [Test] peer_send_me()
INFO 2021-02-23 00:54:24,456 [shard 0] ms - [2] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@49730] intercepted HELLO_READ(3)
INFO 2021-02-23 00:54:24,473 [shard 0] ms - [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@49730] UPDATE: peer_type=osd, policy(lossy=true server=true standby=false resetcheck=false)
INFO 2021-02-23 00:54:24,504 [shard 0] ms - [TestPeer] send_peer()
INFO 2021-02-23 00:54:24,506 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4@49730 >> osd.? v2:127.0.0.1:9012/2] write_event: delay ...
INFO 2021-02-23 00:54:24,517 [shard 0] ms - [2] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@49730] intercepted AUTH_REQUEST_READ(3)
INFO 2021-02-23 00:54:24,546 [shard 0] ms - [2] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@49730] intercepted AUTH_DONE_WRITE(3)
INFO 2021-02-23 00:54:24,554 [shard 0] ms - [2] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@49730] intercepted AUTH_SIGNATURE_WRITE(3)
ERROR 2021-02-23 00:54:24,575 [shard 0] ms - test_messenger timeout after 120s, abort! Consider to extend the period if the test is still running.
INFO 2021-02-23 00:54:24,603 [shard 0] ms - [Test] wait_result(3) ...
INFO 2021-02-23 00:54:24,604 [shard 0] ms - [Test] wait_ready(): wait for connections, currently 3 out of 3, pending 1 ready ...
INFO 2021-02-23 00:54:24,605 [shard 0] ms - [Test] wait_ready(): wait for pending_send=0, pending_peer_receive=0, pending_receive=1, pending 1/0 ready/establish connections ...
unittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
==5750==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004b90 at pc 0x56219b904a18 bp 0x7fff10693da0 sp 0x7fff10693d90
READ of size 8 at 0x619000004b90 thread T0
#0 0x56219b904a17 in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x56219b90ced7 in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x56219b90cbb1 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x56219b90c5b5 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x56219b90b957 in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x56219b904c65 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
#6 0x56219b2d4553 in void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/util/backtrace.hh:62
#7 0x56219b2ac735 in seastar::backtrace_buffer::append_backtrace() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:732
#8 0x56219b15d070 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:753
#9 0x56219b15d1e3 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:760
#10 0x56219b1c0f83 in sigabrt_action /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3513
#11 0x56219b1f9b4e in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3495
#12 0x56219b1f9be9 in _FUN /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3491
#13 0x7fe22164f89f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#14 0x7fe21fd96f46 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x3ef46)
#15 0x7fe21fd988b0 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b0)
#16 0x7fe21fd88429 (/lib/x86_64-linux-gnu/libc.so.6+0x30429)
#17 0x7fe21fd884a1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x304a1)
#18 0x56219aa729cb in seastar::sharded<crimson::common::ConfigProxy>::~sharded() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518
#19 0x7fe21fd9b0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
#20 0x7fe21fd9b1e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
#21 0x7fe21fd79b9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
#22 0x5621988af0a9 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger+0xb1a40a9)
0x619000004b90 is located 16 bytes inside of 1024-byte region [0x619000004b80,0x619000004f80)
freed by thread T0 here:
#0 0x7fe2230cfcef in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ecef)
#1 0x56219b91a1e3 in __gnu_cxx::new_allocator<seastar::shared_object>::deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x56219b916b33 in std::allocator_traits<std::allocator<seastar::shared_object> >::deallocate(std::allocator<seastar::shared_object>&, seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x56219b912019 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x56219b90f93c in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x56219b90fb59 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7fe21fd9b0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
previously allocated by thread T0 here:
#0 0x7fe2230ced6f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dd6f)
#1 0x56219b91cf6e in __gnu_cxx::new_allocator<seastar::shared_object>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x56219b91b9b9 in std::allocator_traits<std::allocator<seastar::shared_object> >::allocate(std::allocator<seastar::shared_object>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x56219b919cff in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x56219b916220 in void std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_realloc_insert<seastar::shared_object>(__gnu_cxx::__normal_iterator<seastar::shared_object*, std::vector<seastar::shared_object, std::allocator<seastar::shared_object> > >, seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:440
#5 0x56219b911dbb in seastar::shared_object& std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::emplace_back<seastar::shared_object>(seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:121
#6 0x56219b90f710 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::push_back(seastar::shared_object&&) /usr/include/c++/9/bits/stl_vector.h:1201
#7 0x56219b904619 in dl_iterate_phdr_callback /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:49
#8 0x7fe21febe160 in dl_iterate_phdr (/lib/x86_64-linux-gnu/libc.so.6+0x166160)
SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76 in operator()
Shadow bytes around the buggy address:
0x0c327fff8920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8950: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff8960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff8970: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8990: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==5750==ABORTING
Updated by Josh Durgin about 3 years ago
unittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
=================================================================
==26366==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004b90 at pc 0x55617cbbda18 bp 0x7ffcdeb080a0 sp 0x7ffcdeb08090
READ of size 8 at 0x619000004b90 thread T0
#0 0x55617cbbda17 in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x55617cbc5ed7 in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x55617cbc5bb1 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x55617cbc55b5 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x55617cbc4957 in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x55617cbbdc65 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
#6 0x55617c58d553 in void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/util/backtrace.hh:62
#7 0x55617c565735 in seastar::backtrace_buffer::append_backtrace() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:732
#8 0x55617c416070 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:753
#9 0x55617c4161e3 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:760
#10 0x55617c479f83 in sigabrt_action /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3513
#11 0x55617c4b2b4e in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3495
#12 0x55617c4b2be9 in _FUN /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3491
#13 0x7f5cce71389f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#14 0x7f5ccce5af46 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x3ef46)
#15 0x7f5ccce5c8b0 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b0)
#16 0x7f5ccce4c429 (/lib/x86_64-linux-gnu/libc.so.6+0x30429)
#17 0x7f5ccce4c4a1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x304a1)
#18 0x55617bd2b9cb in seastar::sharded<crimson::common::ConfigProxy>::~sharded() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518
#19 0x7f5ccce5f0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
#20 0x7f5ccce5f1e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
#21 0x7f5ccce3db9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
#22 0x556179b680a9 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger+0xb1a40a9)
0x619000004b90 is located 16 bytes inside of 1024-byte region [0x619000004b80,0x619000004f80)
freed by thread T0 here:
#0 0x7f5cd0193cef in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ecef)
#1 0x55617cbd31e3 in __gnu_cxx::new_allocator<seastar::shared_object>::deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x55617cbcfb33 in std::allocator_traits<std::allocator<seastar::shared_object> >::deallocate(std::allocator<seastar::shared_object>&, seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x55617cbcb019 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x55617cbc893c in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x55617cbc8b59 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7f5ccce5f0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
previously allocated by thread T0 here:
#0 0x7f5cd0192d6f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dd6f)
#1 0x55617cbd5f6e in __gnu_cxx::new_allocator<seastar::shared_object>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x55617cbd49b9 in std::allocator_traits<std::allocator<seastar::shared_object> >::allocate(std::allocator<seastar::shared_object>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x55617cbd2cff in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x55617cbcf220 in void std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_realloc_insert<seastar::shared_object>(__gnu_cxx::__normal_iterator<seastar::shared_object*, std::vector<seastar::shared_object, std::allocator<seastar::shared_object> > >, seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:440
#5 0x55617cbcadbb in seastar::shared_object& std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::emplace_back<seastar::shared_object>(seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:121
#6 0x55617cbc8710 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::push_back(seastar::shared_object&&) /usr/include/c++/9/bits/stl_vector.h:1201
#7 0x55617cbbd619 in dl_iterate_phdr_callback /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:49
#8 0x7f5cccf82160 in dl_iterate_phdr (/lib/x86_64-linux-gnu/libc.so.6+0x166160)
SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76 in operator()
Shadow bytes around the buggy address:
0x0c327fff8920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8950: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff8960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff8970: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8990: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==26366==ABORTING
Updated by Sebastian Wagner about 3 years ago
INFO 2021-02-23 17:43:59,258 [shard 0] ms - -- 0 --
INFO 2021-02-23 17:43:59,258 [shard 0] ms - [Test] setup connection...
INFO 2021-02-23 17:43:59,258 [shard 0] ms - [Test] peer_connect_me(v2:127.0.0.1:9012/2)
INFO 2021-02-23 17:43:59,271 [shard 0] ms - [TestPeer] connect_peer(v2:127.0.0.1:9012/2)
INFO 2021-02-23 17:43:59,272 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4 >> osd.? v2:127.0.0.1:9012/2] ProtocolV2::start_connect(): peer_addr=v2:127.0.0.1:9012/2, peer_name=osd.?, cc=1286576077112908087 policy(lossy=true, server=false, standby=false, resetcheck=false)
INFO 2021-02-23 17:43:59,273 [shard 0] ms - [TestPeer] this is a new session
INFO 2021-02-23 17:43:59,281 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -] new connection registered
INFO 2021-02-23 17:43:59,281 [shard 0] ms - [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] ProtocolV2::start_accept(): target_addr=127.0.0.1:57769/0
INFO 2021-02-23 17:43:59,282 [shard 0] ms - [Test] accept_attempts=1
INFO 2021-02-23 17:43:59,283 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted SOCKET_ACCEPTED(1)
INFO 2021-02-23 17:43:59,284 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted BANNER_WRITE(1)
INFO 2021-02-23 17:43:59,291 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted BANNER_READ(1)
INFO 2021-02-23 17:43:59,312 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted BANNER_PAYLOAD_READ(1)
INFO 2021-02-23 17:43:59,325 [shard 0] ms - [Test] send_peer() (pending 1)
INFO 2021-02-23 17:43:59,327 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted HELLO_WRITE(1)
INFO 2021-02-23 17:43:59,328 [shard 0] ms - [Test] peer_send_me()
INFO 2021-02-23 17:43:59,341 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> unknown.? -@57769] intercepted HELLO_READ(1)
INFO 2021-02-23 17:43:59,354 [shard 0] ms - [TestPeer] send_peer()
INFO 2021-02-23 17:43:59,354 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4@57769 >> osd.? v2:127.0.0.1:9012/2] write_event: delay ...
INFO 2021-02-23 17:43:59,359 [shard 0] ms - [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] UPDATE: peer_type=osd, policy(lossy=true server=true standby=false resetcheck=false)
INFO 2021-02-23 17:43:59,370 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] intercepted AUTH_REQUEST_READ(1)
INFO 2021-02-23 17:43:59,378 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] intercepted AUTH_DONE_WRITE(1)
INFO 2021-02-23 17:43:59,382 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] intercepted AUTH_SIGNATURE_WRITE(1)
INFO 2021-02-23 17:43:59,387 [shard 0] ms - [Test] wait_result(1) ...
INFO 2021-02-23 17:43:59,388 [shard 0] ms - [Test] wait_ready(): wait for connections, currently 1 out of 1, pending 1 ready ...
INFO 2021-02-23 17:43:59,388 [shard 0] ms - [Test] wait_ready(): wait for pending_send=1, pending_peer_receive=0, pending_receive=1, pending 1/0 ready/establish connections ...
INFO 2021-02-23 17:43:59,404 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] intercepted AUTH_SIGNATURE_READ(1)
INFO 2021-02-23 17:43:59,414 [shard 0] ms - [Test] server_connect_attemps=1
INFO 2021-02-23 17:43:59,414 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.? -@57769] intercepted CLIENT_IDENT_READ(1)
INFO 2021-02-23 17:43:59,419 [shard 0] ms - [Test] got accept (cnt_accept_dispatched=1), track [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769]
INFO 2021-02-23 17:43:59,420 [shard 0] ms - [Test] flush sending 1 ops
INFO 2021-02-23 17:43:59,420 [shard 0] ms - [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] write_event: delay ...
INFO 2021-02-23 17:43:59,423 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] intercepted SERVER_IDENT_WRITE(1)
INFO 2021-02-23 17:43:59,426 [shard 0] ms - [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] established: gs=0, pgs=1, cs=0, client_cookie=1286576077112908087, server_cookie=0, in_seq=0, out_seq=0, out_q=1
INFO 2021-02-23 17:43:59,427 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] ready
INFO 2021-02-23 17:43:59,433 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] intercepted MESSAGE_WRITE(1)
INFO 2021-02-23 17:43:59,435 [shard 0] ms - [Test] wait_ready(): wait for pending_send=0, pending_peer_receive=1, pending_receive=1, pending 0/1 ready/establish connections ...
INFO 2021-02-23 17:43:59,442 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4@57769 >> osd.2 v2:127.0.0.1:9012/2] connected: gs=1, pgs=1, cs=0, client_cookie=1286576077112908087, server_cookie=0, in_seq=0, out_seq=0, out_q=1
INFO 2021-02-23 17:43:59,450 [shard 0] ms - [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769] intercepted MESSAGE_READ(1)
INFO 2021-02-23 17:43:59,459 [shard 0] ms - [Test] got op, left 0 ops -- [0] [osd.2(Test) v2:127.0.0.1:9012/2 >> osd.4 v2:127.0.0.1:9014/4@57769]
INFO 2021-02-23 17:43:59,462 [shard 0] ms - [Test] wait_ready(): wait for pending_send=0, pending_peer_receive=1, pending_receive=0, pending 0/1 ready/establish connections ...
INFO 2021-02-23 17:43:59,465 [shard 0] ms - [TestPeer] got op from Test
INFO 2021-02-23 17:43:59,476 [shard 0] ms - [Test] TestPeer said got op, left 0 ops
INFO 2021-02-23 17:43:59,478 [shard 0] ms - [Test] wait_ready(): wait done!
INFO 2021-02-23 17:43:59,480 [shard 0] ms - -- 1 --
INFO 2021-02-23 17:43:59,481 [shard 0] ms - [Test] client markdown...
INFO 2021-02-23 17:43:59,481 [shard 0] ms - [Test] markdown_peer()
ERROR 2021-02-23 17:43:59,491 [shard 0] ms - test_messenger timeout after 120s, abort! Consider to extend the period if the test is still running.
INFO 2021-02-23 17:43:59,496 [shard 0] ms - [TestPeer] markdown()
INFO 2021-02-23 17:43:59,497 [shard 0] ms - [osd.4(TestPeer) v2:127.0.0.1:9014/4@57769 >> osd.2 v2:127.0.0.1:9012/2] closing: reset no, replace no
unittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
=================================================================
==25808==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004b90 at pc 0x55a1409d6a48 bp 0x7ffc5c40f460 sp 0x7ffc5c40f450
READ of size 8 at 0x619000004b90 thread T0
#0 0x55a1409d6a47 in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x55a1409def07 in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x55a1409debe1 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x55a1409de5e5 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x55a1409dd987 in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x55a1409d6c95 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
#6 0x55a1403a6583 in void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/util/backtrace.hh:62
#7 0x55a14037e765 in seastar::backtrace_buffer::append_backtrace() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:732
#8 0x55a14022f0a0 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:753
#9 0x55a14022f213 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:760
#10 0x55a140292fb3 in sigabrt_action /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3513
#11 0x55a1402cbb7e in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3495
#12 0x55a1402cbc19 in _FUN /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3491
#13 0x7f555212f89f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#14 0x7f5550876f46 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x3ef46)
#15 0x7f55508788b0 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b0)
#16 0x7f5550868429 (/lib/x86_64-linux-gnu/libc.so.6+0x30429)
#17 0x7f55508684a1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x304a1)
#18 0x55a13fb446f1 in seastar::sharded<crimson::common::ConfigProxy>::~sharded() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518
#19 0x7f555087b0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
#20 0x7f555087b1e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
#21 0x7f5550859b9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
#22 0x55a13d98fb59 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger+0xb135b59)
0x619000004b90 is located 16 bytes inside of 1024-byte region [0x619000004b80,0x619000004f80)
freed by thread T0 here:
#0 0x7f5553bafcef in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ecef)
#1 0x55a1409ec213 in __gnu_cxx::new_allocator<seastar::shared_object>::deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x55a1409e8b63 in std::allocator_traits<std::allocator<seastar::shared_object> >::deallocate(std::allocator<seastar::shared_object>&, seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x55a1409e4049 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x55a1409e196c in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x55a1409e1b89 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7f555087b0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
previously allocated by thread T0 here:
#0 0x7f5553baed6f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dd6f)
#1 0x55a1409eef9e in __gnu_cxx::new_allocator<seastar::shared_object>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x55a1409ed9e9 in std::allocator_traits<std::allocator<seastar::shared_object> >::allocate(std::allocator<seastar::shared_object>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x55a1409ebd2f in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x55a1409e8250 in void std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_realloc_insert<seastar::shared_object>(__gnu_cxx::__normal_iterator<seastar::shared_object*, std::vector<seastar::shared_object, std::allocator<seastar::shared_object> > >, seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:440
#5 0x55a1409e3deb in seastar::shared_object& std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::emplace_back<seastar::shared_object>(seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:121
#6 0x55a1409e1740 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::push_back(seastar::shared_object&&) /usr/include/c++/9/bits/stl_vector.h:1201
#7 0x55a1409d6649 in dl_iterate_phdr_callback /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:49
#8 0x7f555099e160 in dl_iterate_phdr (/lib/x86_64-linux-gnu/libc.so.6+0x166160)
SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76 in operator()
Shadow bytes around the buggy address:
0x0c327fff8920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8950: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff8960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff8970: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8990: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff89c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==25808==ABORTING
Updated by Kefu Chai about 3 years ago
all of these failed tests were running on 172.21.3.232+irvingi03. i think they are caused by one or more seastar tests, which were competing with unittest-seastar-messenger for CPU cycles.
i just logged onto the test node and killed all seastar tests.
Updated by Kefu Chai about 3 years ago
- Is duplicate of Bug #47830: crimson unit tests never end added
Actions