Bug #43433
open[ceph-mon]ceph-mon crash when modify pool's attribute.
0%
Description
os: centos 7.6.1810
ceph version: 13.2.5
Hardware: Dell Inc PowerEdge R730.
Steps:
The ceph-mon crash When modify pool's attribute.
Calltrace:
#0 0x00007fa4beded49b in raise () from /lib64/libpthread.so.0
#1 0x000055fd20c4f211 in reraise_fatal (signum=11) at /usr/src/debug/ceph-13.2.5-1.0.9/src/global/signal_handler.cc:74
#2 handle_fatal_signal (signum=11) at /usr/src/debug/ceph-13.2.5-1.0.9/src/global/signal_handler.cc:138
#3 <signal handler called>
#4 SLL_Next (t=0x55fd0000000b) at src/linked_list.h:45
#5 SLL_TryPop (rv=<synthetic pointer>, list=0x55fd227d44a0) at src/linked_list.h:69
#6 TryPop (rv=<synthetic pointer>, this=0x55fd227d44a0) at src/thread_cache.h:220
#7 Allocate (cl=43, size=4096, this=<optimized out>) at src/thread_cache.h:381
#8 do_malloc (size=4096) at src/tcmalloc.cc:1289
#9 (anonymous namespace)::do_memalign (align=align@entry=8, size=size@entry=4096) at src/tcmalloc.cc:1509
#10 0x00007fa4be187010 in do_memalign_or_cpp_memalign (size=4096, align=8) at src/tcmalloc.cc:1215
#11 tc_posix_memalign (result_ptr=result_ptr@entry=0x7fa4b1d4c3b0, align=align@entry=8, size=size@entry=4096) at src/tcmalloc.cc:1948
#12 0x00007fa4bfa51de6 in create (mempool=10, align=8, len=4024) at /usr/src/debug/ceph-13.2.5-1.0.9/src/common/buffer.cc:307
#13 ceph::buffer::list::append (this=this@entry=0x55fd334d4f28, data=data@entry=0x55fd2b3cab58 <incomplete sequence \372>, len=len@entry=53) at /usr/src/debug/ceph-13.2.5-1.0.9/src/common/buffer.cc:1959
#14 0x00007fa4bfb62398 in encode_raw<ceph_msg_header> (bl=..., t=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/include/encoding.h:74
#15 encode (features=0, bl=..., v=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/include/types.h:305
#16 encode_message (msg=msg@entry=0x55fd2b3cab40, features=features@entry=4611087854031667195, payload=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/msg/Message.cc:945
#17 0x000055fd20a5d1f2 in Monitor::forward_request_leader (this=0x55fd234c4000, op=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/mon/Monitor.cc:3643
#18 0x000055fd20b5e2d9 in PaxosService::dispatch (this=0x55fd234c5400, op=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/mon/PaxosService.cc:79
#19 0x000055fd20a73e98 in Monitor::dispatch_op (this=this@entry=0x55fd234c4000, op=...) at /usr/src/debug/ceph-13.2.5-1.0.9/src/mon/Monitor.cc:4166
#20 0x000055fd20a75342 in Monitor::_ms_dispatch (this=this@entry=0x55fd234c4000, m=m@entry=0x55fd2b3cab40) at /usr/src/debug/ceph-13.2.5-1.0.9/src/mon/Monitor.cc:4085
#21 0x000055fd20a9ae23 in Monitor::ms_dispatch (this=0x55fd234c4000, m=0x55fd2b3cab40) at /usr/src/debug/ceph-13.2.5-1.0.9/src/mon/Monitor.h:880
#22 0x00007fa4bfb5f39a in ms_deliver_dispatch (m=0x55fd2b3cab40, this=0x55fd22fe6e00) at /usr/src/debug/ceph-13.2.5-1.0.9/src/msg/Messenger.h:667
#23 DispatchQueue::entry (this=0x55fd22fe6f58) at /usr/src/debug/ceph-13.2.5-1.0.9/src/msg/DispatchQueue.cc:201
#24 0x00007fa4bfbfd2cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /usr/src/debug/ceph-13.2.5-1.0.9/src/msg/DispatchQueue.h:101
#25 0x00007fa4bede5dd5 in start_thread () from /lib64/libpthread.so.0
#26 0x00007fa4bb701ead in clone () from /lib64/libc.so.6
Updated by Joao Eduardo Luis over 4 years ago
Can you please share the command that triggered the crash?
Updated by haitao chen over 4 years ago
Joao Eduardo Luis wrote:
Can you please share the command that triggered the crash?
Open the core dump config of the centos.
It will generated a core file. Install ceph-debuginfo.rpm and gperftools-debuginfo.rpm.
Use gdb -c corefile. And then use bt command to show the call trace.
Updated by haitao chen over 4 years ago
It maybe double free or write data into a free memory.