Project

General

Profile

Bug #48108 ยป msgr.log

Kefu Chai, 11/10/2020 09:19 AM

 
190/211 Test #198: unittest-seastar-messenger ................***Failed 52.88 sec
WARNING: debug mode. Not for benchmarking or production
WARN 2020-11-10 07:16:09,070 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
INFO 2020-11-10 07:16:09,295 [shard 0] ms - test_echo(rounds=512, keepalive_ratio=0.1, v2=false):
INFO 2020-11-10 07:16:09,303 [shard 0] ms - [osd.1(server2) v1:127.0.0.1:9011/2] bind: done
INFO 2020-11-10 07:16:09,305 [shard 0] ms - [osd.0(server1) v1:127.0.0.1:9010/1] bind: done
INFO 2020-11-10 07:16:09,310 [shard 0] ms - [osd.3(client2) - >> osd.? v1:127.0.0.1:9010/1] write_event: delay ...
INFO 2020-11-10 07:16:09,312 [shard 0] ms - [osd.2(client1) - >> osd.? v1:127.0.0.1:9011/2] write_event: delay ...
INFO 2020-11-10 07:16:09,336 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1] learned myaddr=127.0.0.1:0/4 (unbound) from v1:127.0.0.1:56682/0
INFO 2020-11-10 07:16:09,339 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2] learned myaddr=127.0.0.1:0/3 (unbound) from v1:127.0.0.1:51151/0
INFO 2020-11-10 07:16:10,667 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1]: finished sending 512 pings with 47 keepalives
INFO 2020-11-10 07:16:10,697 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2]: finished sending 512 pings with 68 keepalives
INFO 2020-11-10 07:16:18,898 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1]: finished receiving 512 pongs
INFO 2020-11-10 07:16:18,899 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2]: finished receiving 512 pongs
INFO 2020-11-10 07:16:18,902 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2]: handshake 0.071277716, pingpong 9.518251289
INFO 2020-11-10 07:16:18,902 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1]: handshake 0.07030454, pingpong 9.521988856
INFO 2020-11-10 07:16:18,905 [shard 0] ms - client1 shutdown...
INFO 2020-11-10 07:16:18,906 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2] closing: reset no, replace no
WARN 2020-11-10 07:16:18,914 [shard 0] ms - [osd.1(server2) v1:127.0.0.1:9011/2 >> osd.? v1:127.0.0.1:0/3@51151] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:18,914 [shard 0] ms - [osd.1(server2) v1:127.0.0.1:9011/2 >> osd.? v1:127.0.0.1:0/3@51151] closing: reset yes, replace no
WARN 2020-11-10 07:16:18,915 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@51151 >> osd.? v1:127.0.0.1:9011/2] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:18,922 [shard 0] ms - client2 shutdown...
INFO 2020-11-10 07:16:18,924 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1] closing: reset no, replace no
WARN 2020-11-10 07:16:18,929 [shard 0] ms - [osd.0(server1) v1:127.0.0.1:9010/1 >> osd.? v1:127.0.0.1:0/4@56682] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:18,930 [shard 0] ms - [osd.0(server1) v1:127.0.0.1:9010/1 >> osd.? v1:127.0.0.1:0/4@56682] closing: reset yes, replace no
WARN 2020-11-10 07:16:18,933 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@56682 >> osd.? v1:127.0.0.1:9010/1] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:18,940 [shard 0] ms - server1 shutdown...
INFO 2020-11-10 07:16:18,947 [shard 0] ms - server2 shutdown...
INFO 2020-11-10 07:16:18,954 [shard 0] ms - test_echo() done!

INFO 2020-11-10 07:16:18,955 [shard 0] ms - test_echo(rounds=512, keepalive_ratio=0.1, v2=true):
INFO 2020-11-10 07:16:18,963 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1] bind: done
INFO 2020-11-10 07:16:18,964 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2] bind: done
INFO 2020-11-10 07:16:18,967 [shard 0] ms - [osd.3(client2) - >> osd.? v2:127.0.0.1:9010/1] ProtocolV2::start_connect(): peer_addr=v2:127.0.0.1:9010/1, peer_name=osd.?, cc=12409947247928050800 policy(lossy=true, server=false, standby=false, resetcheck=false)
INFO 2020-11-10 07:16:18,986 [shard 0] ms - [osd.3(client2) - >> osd.? v2:127.0.0.1:9010/1] write_event: delay ...
INFO 2020-11-10 07:16:18,986 [shard 0] ms - [osd.2(client1) - >> osd.? v2:127.0.0.1:9011/2] ProtocolV2::start_connect(): peer_addr=v2:127.0.0.1:9011/2, peer_name=osd.?, cc=11253588019763930121 policy(lossy=true, server=false, standby=false, resetcheck=false)
INFO 2020-11-10 07:16:18,987 [shard 0] ms - [osd.2(client1) - >> osd.? v2:127.0.0.1:9011/2] write_event: delay ...
INFO 2020-11-10 07:16:18,988 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1 >> unknown.? -@54786] ProtocolV2::start_accept(): target_addr=127.0.0.1:54786/0
INFO 2020-11-10 07:16:18,992 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2 >> unknown.? -@64111] ProtocolV2::start_accept(): target_addr=127.0.0.1:64111/0
INFO 2020-11-10 07:16:19,049 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1 >> osd.? -@54786] UPDATE: peer_type=osd, policy(lossy=true server=true standby=false resetcheck=false)
INFO 2020-11-10 07:16:19,053 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.? v2:127.0.0.1:9010/1] learned myaddr=127.0.0.1:0/4 (unbound) from v2:127.0.0.1:54786/0
INFO 2020-11-10 07:16:19,056 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2 >> osd.? -@64111] UPDATE: peer_type=osd, policy(lossy=true server=true standby=false resetcheck=false)
INFO 2020-11-10 07:16:19,069 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.? v2:127.0.0.1:9011/2] learned myaddr=127.0.0.1:0/3 (unbound) from v2:127.0.0.1:64111/0
INFO 2020-11-10 07:16:19,157 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1 >> osd.3 127.0.0.1:0/4@54786] established: gs=0, pgs=1, cs=0, client_cookie=12409947247928050800, server_cookie=0, in_seq=0, out_seq=0, out_q=0
INFO 2020-11-10 07:16:19,168 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1] connected: gs=1, pgs=1, cs=0, client_cookie=12409947247928050800, server_cookie=0, in_seq=0, out_seq=0, out_q=65
INFO 2020-11-10 07:16:19,197 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2 >> osd.2 127.0.0.1:0/3@64111] established: gs=0, pgs=1, cs=0, client_cookie=11253588019763930121, server_cookie=0, in_seq=0, out_seq=0, out_q=0
INFO 2020-11-10 07:16:19,221 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2] connected: gs=1, pgs=1, cs=0, client_cookie=11253588019763930121, server_cookie=0, in_seq=0, out_seq=0, out_q=71
INFO 2020-11-10 07:16:23,508 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1]: finished sending 512 pings with 52 keepalives
INFO 2020-11-10 07:16:23,740 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2]: finished sending 512 pings with 62 keepalives
INFO 2020-11-10 07:16:53,304 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2]: finished receiving 512 pongs
INFO 2020-11-10 07:16:53,311 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2]: handshake 0.236133366, pingpong 34.083367597
INFO 2020-11-10 07:16:53,329 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1]: finished receiving 512 pongs
INFO 2020-11-10 07:16:53,334 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1]: handshake 0.202266829, pingpong 34.161519905
INFO 2020-11-10 07:16:53,338 [shard 0] ms - client1 shutdown...
INFO 2020-11-10 07:16:53,340 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2] closing: reset no, replace no
INFO 2020-11-10 07:16:53,351 [shard 0] ms - [osd.2(client1) 127.0.0.1:0/3@64111 >> osd.1 v2:127.0.0.1:9011/2] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,361 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2 >> osd.2 127.0.0.1:0/3@64111] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,364 [shard 0] ms - [osd.1(server2) v2:127.0.0.1:9011/2 >> osd.2 127.0.0.1:0/3@64111] closing: reset yes, replace no
INFO 2020-11-10 07:16:53,384 [shard 0] ms - client2 shutdown...
INFO 2020-11-10 07:16:53,386 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1] closing: reset no, replace no
INFO 2020-11-10 07:16:53,396 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1 >> osd.3 127.0.0.1:0/4@54786] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,399 [shard 0] ms - [osd.0(server1) v2:127.0.0.1:9010/1 >> osd.3 127.0.0.1:0/4@54786] closing: reset yes, replace no
INFO 2020-11-10 07:16:53,407 [shard 0] ms - [osd.3(client2) 127.0.0.1:0/4@54786 >> osd.0 v2:127.0.0.1:9010/1] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,424 [shard 0] ms - server1 shutdown...
INFO 2020-11-10 07:16:53,441 [shard 0] ms - server2 shutdown...
INFO 2020-11-10 07:16:53,457 [shard 0] ms - test_echo() done!

INFO 2020-11-10 07:16:53,460 [shard 0] ms - test_concurrent_dispatch(v2=false):
INFO 2020-11-10 07:16:53,472 [shard 0] ms - [osd.4(server3) v1:127.0.0.1:9010/5] bind: done
INFO 2020-11-10 07:16:53,482 [shard 0] ms - [osd.5(client3) - >> osd.? v1:127.0.0.1:9010/5] write_event: delay ...
INFO 2020-11-10 07:16:53,512 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@50794 >> osd.? v1:127.0.0.1:9010/5] learned myaddr=127.0.0.1:0/6 (unbound) from v1:127.0.0.1:50794/0
INFO 2020-11-10 07:16:53,608 [shard 0] ms - client shutdown...
INFO 2020-11-10 07:16:53,611 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@50794 >> osd.? v1:127.0.0.1:9010/5] closing: reset no, replace no
WARN 2020-11-10 07:16:53,617 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@50794 >> osd.? v1:127.0.0.1:9010/5] open fault: std::system_error (error crimson::net:4, read eof)
WARN 2020-11-10 07:16:53,625 [shard 0] ms - [osd.4(server3) v1:127.0.0.1:9010/5 >> osd.? v1:127.0.0.1:0/6@50794] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,627 [shard 0] ms - [osd.4(server3) v1:127.0.0.1:9010/5 >> osd.? v1:127.0.0.1:0/6@50794] closing: reset yes, replace no
INFO 2020-11-10 07:16:53,639 [shard 0] ms - server shutdown...
INFO 2020-11-10 07:16:53,653 [shard 0] ms - test_concurrent_dispatch() done!

INFO 2020-11-10 07:16:53,655 [shard 0] ms - test_concurrent_dispatch(v2=true):
INFO 2020-11-10 07:16:53,663 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5] bind: done
INFO 2020-11-10 07:16:53,669 [shard 0] ms - [osd.5(client3) - >> osd.? v2:127.0.0.1:9010/5] ProtocolV2::start_connect(): peer_addr=v2:127.0.0.1:9010/5, peer_name=osd.?, cc=14292136439070309660 policy(lossy=true, server=false, standby=false, resetcheck=false)
INFO 2020-11-10 07:16:53,672 [shard 0] ms - [osd.5(client3) - >> osd.? v2:127.0.0.1:9010/5] write_event: delay ...
INFO 2020-11-10 07:16:53,680 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5 >> unknown.? -@49418] ProtocolV2::start_accept(): target_addr=127.0.0.1:49418/0
INFO 2020-11-10 07:16:53,736 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5 >> osd.? -@49418] UPDATE: peer_type=osd, policy(lossy=true server=true standby=false resetcheck=false)
INFO 2020-11-10 07:16:53,744 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@49418 >> osd.? v2:127.0.0.1:9010/5] learned myaddr=127.0.0.1:0/6 (unbound) from v2:127.0.0.1:49418/0
INFO 2020-11-10 07:16:53,824 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5 >> osd.5 127.0.0.1:0/6@49418] established: gs=0, pgs=1, cs=0, client_cookie=14292136439070309660, server_cookie=0, in_seq=0, out_seq=0, out_q=0
INFO 2020-11-10 07:16:53,839 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@49418 >> osd.4 v2:127.0.0.1:9010/5] connected: gs=1, pgs=1, cs=0, client_cookie=14292136439070309660, server_cookie=0, in_seq=0, out_seq=0, out_q=2
INFO 2020-11-10 07:16:53,873 [shard 0] ms - client shutdown...
INFO 2020-11-10 07:16:53,875 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@49418 >> osd.4 v2:127.0.0.1:9010/5] closing: reset no, replace no
INFO 2020-11-10 07:16:53,885 [shard 0] ms - [osd.5(client3) 127.0.0.1:0/6@49418 >> osd.4 v2:127.0.0.1:9010/5] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,889 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5 >> osd.5 127.0.0.1:0/6@49418] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:53,892 [shard 0] ms - [osd.4(server3) v2:127.0.0.1:9010/5 >> osd.5 127.0.0.1:0/6@49418] closing: reset yes, replace no
INFO 2020-11-10 07:16:53,909 [shard 0] ms - server shutdown...
INFO 2020-11-10 07:16:53,925 [shard 0] ms - test_concurrent_dispatch() done!

INFO 2020-11-10 07:16:53,926 [shard 0] ms - test_preemptive_shutdown(v2=false):
INFO 2020-11-10 07:16:53,936 [shard 0] ms - [osd.6(server4) v1:127.0.0.1:9010/7] bind: done
INFO 2020-11-10 07:16:53,942 [shard 0] ms - [osd.7(client4) - >> osd.? v1:127.0.0.1:9010/7] write_event: delay ...
INFO 2020-11-10 07:16:53,998 [shard 0] ms - [osd.7(client4) 127.0.0.1:0/8@57764 >> osd.? v1:127.0.0.1:9010/7] learned myaddr=127.0.0.1:0/8 (unbound) from v1:127.0.0.1:57764/0
INFO 2020-11-10 07:16:54,047 [shard 0] ms - client shutdown...
INFO 2020-11-10 07:16:54,059 [shard 0] ms - [osd.7(client4) 127.0.0.1:0/8@57764 >> osd.? v1:127.0.0.1:9010/7] closing: reset no, replace no
INFO 2020-11-10 07:16:54,067 [shard 0] ms - [osd.7(client4) 127.0.0.1:0/8@57764 >> osd.? v1:127.0.0.1:9010/7] write_event: dropped
WARN 2020-11-10 07:16:54,076 [shard 0] ms - [osd.6(server4) v1:127.0.0.1:9010/7 >> osd.? v1:127.0.0.1:0/8@57764] open fault: std::system_error (error crimson::net:4, read eof)
INFO 2020-11-10 07:16:54,078 [shard 0] ms - [osd.6(server4) v1:127.0.0.1:9010/7 >> osd.? v1:127.0.0.1:0/8@57764] closing: reset yes, replace no
WARN 2020-11-10 07:16:54,089 [shard 0] ms - [osd.7(client4) 127.0.0.1:0/8@57764 >> osd.? v1:127.0.0.1:9010/7] connecting fault: seastar::gate_closed_exception (gate closed)
INFO 2020-11-10 07:16:54,104 [shard 0] ms - server shutdown...
INFO 2020-11-10 07:16:54,116 [shard 0] ms - test_preemptive_shutdown() done!

Test failure
ERROR 2020-11-10 07:16:54,120 [shard 0] seastar - Exiting on unhandled exception: seastar::gate_closed_exception (gate closed)
unittest-seastar-messenger: /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518: seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed.
=================================================================
==7818==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004690 at pc 0x55e322547cfc bp 0x7ffc8fd29b20 sp 0x7ffc8fd29b10
READ of size 8 at 0x619000004690 thread T0
#0 0x55e322547cfb in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76
#1 0x55e3225501bb in operator()<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> > > /usr/include/c++/9/bits/predefined_ops.h:283
#2 0x55e32254fe95 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:120
#3 0x55e32254f899 in __find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, __gnu_cxx::__ops::_Iter_pred<seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > > /usr/include/c++/9/bits/stl_algo.h:161
#4 0x55e32254ec3b in find_if<__gnu_cxx::__normal_iterator<const seastar::shared_object*, std::vector<seastar::shared_object> >, seastar::decorate(uintptr_t)::<lambda(const seastar::shared_object&)> > /usr/include/c++/9/bits/stl_algo.h:3963
#5 0x55e322547f49 in seastar::decorate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:77
#6 0x55e321f1a56b in void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/util/backtrace.hh:62
#7 0x55e321ef2677 in seastar::backtrace_buffer::append_backtrace() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:731
#8 0x55e321da7990 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:752
#9 0x55e321da7b03 in print_with_backtrace /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:759
#10 0x55e321e09cbb in sigabrt_action /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3486
#11 0x55e321e4180a in operator() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3468
#12 0x55e321e418a5 in _FUN /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/core/reactor.cc:3464
#13 0x7f6f5188289f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#14 0x7f6f4ffc9f46 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x3ef46)
#15 0x7f6f4ffcb8b0 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b0)
#16 0x7f6f4ffbb429 (/lib/x86_64-linux-gnu/libc.so.6+0x30429)
#17 0x7f6f4ffbb4a1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x304a1)
#18 0x55e32169d85f in seastar::sharded<crimson::common::ConfigProxy>::~sharded() /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include/seastar/core/sharded.hh:518
#19 0x7f6f4ffce0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
#20 0x7f6f4ffce1e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
#21 0x7f6f4ffacb9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
#22 0x55e31f5a4929 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger+0xad73929)

0x619000004690 is located 16 bytes inside of 1024-byte region [0x619000004680,0x619000004a80)
freed by thread T0 here:
#0 0x7f6f53302cef in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ecef)
#1 0x55e32255d4c7 in __gnu_cxx::new_allocator<seastar::shared_object>::deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x55e322559e17 in std::allocator_traits<std::allocator<seastar::shared_object> >::deallocate(std::allocator<seastar::shared_object>&, seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x55e3225552fd in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_deallocate(seastar::shared_object*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x55e322552c20 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x55e322552e3d in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7f6f4ffce0f0 (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)

previously allocated by thread T0 here:
#0 0x7f6f53301d6f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dd6f)
#1 0x55e322560252 in __gnu_cxx::new_allocator<seastar::shared_object>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x55e32255ec9d in std::allocator_traits<std::allocator<seastar::shared_object> >::allocate(std::allocator<seastar::shared_object>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x55e32255cfe3 in std::_Vector_base<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x55e322559504 in void std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::_M_realloc_insert<seastar::shared_object>(__gnu_cxx::__normal_iterator<seastar::shared_object*, std::vector<seastar::shared_object, std::allocator<seastar::shared_object> > >, seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:440
#5 0x55e32255509f in seastar::shared_object& std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::emplace_back<seastar::shared_object>(seastar::shared_object&&) /usr/include/c++/9/bits/vector.tcc:121
#6 0x55e3225529f4 in std::vector<seastar::shared_object, std::allocator<seastar::shared_object> >::push_back(seastar::shared_object&&) /usr/include/c++/9/bits/stl_vector.h:1201
#7 0x55e3225478fd in dl_iterate_phdr_callback /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:49
#8 0x7f6f500f1160 in dl_iterate_phdr (/lib/x86_64-linux-gnu/libc.so.6+0x166160)

SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/src/util/backtrace.cc:76 in operator()
Shadow bytes around the buggy address:
0x0c327fff8880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8890: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff88a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff88b0: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff88c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff88d0: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff88e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff88f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==7818==ABORTING

    (1-1/1)