Bug #61400
Updated by Patrick Donnelly 12 months ago
<pre>
0> 2023-05-24T02:54:54.546+0000 708e7c0 -1 *** Caught signal (Segmentation fault) **
in thread 708e7c0 thread_name:ceph-mon
ceph version 18.0.0-4167-gfa0e62c4 (fa0e62c4a1d8e4a737d9cbe50224f70009b79b28) reef (dev)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109b0dd]
8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109a139]
9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf56d80]
10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5ca1d]
11: main()
12: __libc_start_main()
13: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/ceph-mon.a.log.gz
This happened shortly after the ceph-mon starts.
Here's the error from valgrind as well:
<pre>
<error>
<unique>0x1</unique>
<tid>1</tid>
<threadname>ceph-mon</threadname>
<kind>InvalidRead</kind>
<what>Invalid read of size 8</what>
<stack>
<frame>
<ip>0x55F50A0</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)</fn>
</frame>
<frame>
<ip>0x55F536F</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)</fn>
</frame>
<frame>
<ip>0x55F542F</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)</fn>
</frame>
<frame>
<ip>0x55F8E45</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))</fn>
</frame>
<frame>
<ip>0x5609014</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::allocate_full_cpp_throw_oom(unsigned long)</fn>
</frame>
<frame>
<ip>0x109B0DC</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)</fn>
</frame>
<frame>
<ip>0x109A138</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)</fn>
</frame>
<frame>
<ip>0xF56D7F</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)</fn>
<dir>./obj-x86_64-linux-gnu/src/kv/./src/kv</dir>
<file>RocksDBStore.cc</file>
<line>1193</line>
</frame>
<frame>
<ip>0xC5CA1C</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>MonitorDBStore::open(std::ostream&)</fn>
<dir>./obj-x86_64-linux-gnu/src/./src/mon</dir>
<file>MonitorDBStore.h</file>
<line>674</line>
</frame>
<frame>
<ip>0xC368BF</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>main</fn>
<dir>./obj-x86_64-linux-gnu/src/./src</dir>
<file>ceph_mon.cc</file>
<line>639</line>
</frame>
</stack>
<auxwhat>Address 0x20 is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>
</pre>
From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/valgrind/mon.a.log.gz