Actions
Bug #61400
openvalgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
0> 2023-05-24T02:54:54.546+0000 708e7c0 -1 *** Caught signal (Segmentation fault) ** in thread 708e7c0 thread_name:ceph-mon ceph version 18.0.0-4167-gfa0e62c4 (fa0e62c4a1d8e4a737d9cbe50224f70009b79b28) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420] 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0] 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370] 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430] 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46] 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015] 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109b0dd] 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109a139] 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf56d80] 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5ca1d] 11: main() 12: __libc_start_main() 13: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/ceph-mon.a.log.gz
This happened shortly after the ceph-mon starts.
Here's the error from valgrind as well:
<error> <unique>0x1</unique> <tid>1</tid> <threadname>ceph-mon</threadname> <kind>InvalidRead</kind> <what>Invalid read of size 8</what> <stack> <frame> <ip>0x55F50A0</ip> <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj> <fn>tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)</fn> </frame> <frame> <ip>0x55F536F</ip> <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj> <fn>tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)</fn> </frame> <frame> <ip>0x55F542F</ip> <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj> <fn>tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)</fn> </frame> <frame> <ip>0x55F8E45</ip> <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj> <fn>tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))</fn> </frame> <frame> <ip>0x5609014</ip> <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj> <fn>tcmalloc::allocate_full_cpp_throw_oom(unsigned long)</fn> </frame> <frame> <ip>0x109B0DC</ip> <obj>/usr/bin/ceph-mon</obj> <fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)</fn> </frame> <frame> <ip>0x109A138</ip> <obj>/usr/bin/ceph-mon</obj> <fn>rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)</fn> </frame> <frame> <ip>0xF56D7F</ip> <obj>/usr/bin/ceph-mon</obj> <fn>RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)</fn> <dir>./obj-x86_64-linux-gnu/src/kv/./src/kv</dir> <file>RocksDBStore.cc</file> <line>1193</line> </frame> <frame> <ip>0xC5CA1C</ip> <obj>/usr/bin/ceph-mon</obj> <fn>MonitorDBStore::open(std::ostream&)</fn> <dir>./obj-x86_64-linux-gnu/src/./src/mon</dir> <file>MonitorDBStore.h</file> <line>674</line> </frame> <frame> <ip>0xC368BF</ip> <obj>/usr/bin/ceph-mon</obj> <fn>main</fn> <dir>./obj-x86_64-linux-gnu/src/./src</dir> <file>ceph_mon.cc</file> <line>639</line> </frame> </stack> <auxwhat>Address 0x20 is not stack'd, malloc'd or (recently) free'd</auxwhat> </error>
From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/valgrind/mon.a.log.gz
Updated by Kotresh Hiremath Ravishankar 11 months ago
Updated by Venky Shankar 11 months ago
- Assignee set to Radoslaw Zarzynski
- Severity changed from 3 - minor to 2 - major
Actions