Ceph

     0> 2023-05-24T02:54:54.546+0000 708e7c0 -1 *** Caught signal (Segmentation fault) **
 in thread 708e7c0 thread_name:ceph-mon

 ceph version 18.0.0-4167-gfa0e62c4 (fa0e62c4a1d8e4a737d9cbe50224f70009b79b28) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109b0dd]
 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109a139]
 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf56d80]
 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5ca1d]
 11: main()
 12: __libc_start_main()
 13: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/ceph-mon.a.log.gz

This happened shortly after the ceph-mon starts.

Here's the error from valgrind as well:

<error>
  <unique>0x1</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x55F50A0</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)</fn>
    </frame>
    <frame>
      <ip>0x55F536F</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)</fn>
    </frame>
    <frame>
      <ip>0x55F542F</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)</fn>
    </frame>
    <frame>
      <ip>0x55F8E45</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))</fn>
    </frame>
    <frame>
      <ip>0x5609014</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::allocate_full_cpp_throw_oom(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x109B0DC</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**, bool, bool)</fn>
    </frame>
    <frame>
      <ip>0x109A138</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::DB::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**)</fn>
    </frame>
    <frame>
      <ip>0xF56D7F</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>RocksDBStore::do_open(std::ostream&amp;, bool, bool, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)</fn>
      <dir>./obj-x86_64-linux-gnu/src/kv/./src/kv</dir>
      <file>RocksDBStore.cc</file>
      <line>1193</line>
    </frame>
    <frame>
      <ip>0xC5CA1C</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>MonitorDBStore::open(std::ostream&amp;)</fn>
      <dir>./obj-x86_64-linux-gnu/src/./src/mon</dir>
      <file>MonitorDBStore.h</file>
      <line>674</line>
    </frame>
    <frame>
      <ip>0xC368BF</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>main</fn>
      <dir>./obj-x86_64-linux-gnu/src/./src</dir>
      <file>ceph_mon.cc</file>
      <line>639</line>
    </frame>
  </stack>
  <auxwhat>Address 0x20 is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>

From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/valgrind/mon.a.log.gz