Bug #61820: mon: segfault on rocksdb opening - RADOS - Ceph

Actions

Copy link

Bug #61820

open

mon: segfault on rocksdb opening

Added by Milind Changire 10 months ago. Updated 3 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Matan Breizman

Category:

Target version:

% Done:

Source:

Tags:

backport_processed

Backport:

reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

52539

Crash signature (v1):

Crash signature (v2):

Description

2023-06-23T20:04:55.429 INFO:tasks.ceph:Setting crush tunables to default
2023-06-23T20:04:55.429 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd crush tunables default
.
.
2023-06-23T20:05:05.870 INFO:tasks.ceph.mon.a.smithi133.stderr:==00:00:00:10.291 35389== brk segment overflow in thread #1: can't grow to 0x4830000
2023-06-23T20:05:05.871 INFO:tasks.ceph.mon.a.smithi133.stderr:==00:00:00:10.291 35389== (see section Limitations in user manual)
2023-06-23T20:05:05.871 INFO:tasks.ceph.mon.a.smithi133.stderr:==00:00:00:10.291 35389== NOTE: further instances of this message will not be shown
2023-06-23T20:05:05.879 INFO:tasks.ceph.mon.a.smithi133.stderr:*** Caught signal (Segmentation fault) **
2023-06-23T20:05:05.879 INFO:tasks.ceph.mon.a.smithi133.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:05.953 INFO:tasks.ceph.mon.a.smithi133.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:05.954 INFO:tasks.ceph.mon.a.smithi133.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:05.954 INFO:tasks.ceph.mon.a.smithi133.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:05.955 INFO:tasks.ceph.mon.a.smithi133.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:05.955 INFO:tasks.ceph.mon.a.smithi133.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:05.955 INFO:tasks.ceph.mon.a.smithi133.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:05.956 INFO:tasks.ceph.mon.a.smithi133.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:05.956 INFO:tasks.ceph.mon.a.smithi133.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:05.957 INFO:tasks.ceph.mon.a.smithi133.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:05.958 INFO:tasks.ceph.mon.a.smithi133.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:05.958 INFO:tasks.ceph.mon.a.smithi133.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:05.958 INFO:tasks.ceph.mon.a.smithi133.stderr: 11: main()
2023-06-23T20:05:05.958 INFO:tasks.ceph.mon.a.smithi133.stderr: 12: __libc_start_main()
2023-06-23T20:05:05.959 INFO:tasks.ceph.mon.a.smithi133.stderr: 13: _start()
2023-06-23T20:05:05.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-23T20:05:05.964+0000 7f79e9dca700  1 --  >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79c8005c30 msgr2=0x7f79c80076d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.188:3301/0
2023-06-23T20:05:05.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-23T20:05:05.964+0000 7f79e95c9700  1 --  >> v2:172.21.15.188:3300/0 conn(0x7f79c8007c10 msgr2=0x7f79c8008030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.188:3300/0
2023-06-23T20:05:05.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-23T20:05:05.964+0000 7f79e9dca700  1 --2-  >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79c8005c30 0x7f79c80076d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000
2023-06-23T20:05:05.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-23T20:05:05.964+0000 7f79e95c9700  1 --2-  >> v2:172.21.15.188:3300/0 conn(0x7f79c8007c10 0x7f79c8008030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000
2023-06-23T20:05:05.983 INFO:tasks.ceph.mon.a.smithi133.stderr:2023-06-23T20:05:05.972+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:05.984 INFO:tasks.ceph.mon.a.smithi133.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:05.984 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:05.984 INFO:tasks.ceph.mon.a.smithi133.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:05.984 INFO:tasks.ceph.mon.a.smithi133.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:05.985 INFO:tasks.ceph.mon.a.smithi133.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 11: main()
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 12: __libc_start_main()
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: 13: _start()
2023-06-23T20:05:05.986 INFO:tasks.ceph.mon.a.smithi133.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:05.987 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:05.990 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:00:10.393 35433== brk segment overflow in thread #1: can't grow to 0x4830000
2023-06-23T20:05:05.990 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:00:10.394 35433== (see section Limitations in user manual)
2023-06-23T20:05:05.991 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:00:10.394 35433== NOTE: further instances of this message will not be shown
2023-06-23T20:05:05.991 INFO:tasks.ceph.mon.a.smithi133.stderr:     0> 2023-06-23T20:05:05.972+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:05.992 INFO:tasks.ceph.mon.a.smithi133.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:05.992 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:05.992 INFO:tasks.ceph.mon.a.smithi133.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:05.992 INFO:tasks.ceph.mon.a.smithi133.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:05.992 INFO:tasks.ceph.mon.a.smithi133.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:05.993 INFO:tasks.ceph.mon.a.smithi133.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: 11: main()
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: 12: __libc_start_main()
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: 13: _start()
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:05.994 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:05.998 INFO:tasks.ceph.mon.b.smithi188.stderr:*** Caught signal (Segmentation fault) **
2023-06-23T20:05:05.998 INFO:tasks.ceph.mon.b.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.001 INFO:tasks.ceph.mon.c.smithi188.stderr:==00:00:00:10.406 35434== brk segment overflow in thread #1: can't grow to 0x4830000
2023-06-23T20:05:06.001 INFO:tasks.ceph.mon.c.smithi188.stderr:==00:00:00:10.406 35434== (see section Limitations in user manual)
2023-06-23T20:05:06.001 INFO:tasks.ceph.mon.c.smithi188.stderr:==00:00:00:10.406 35434== NOTE: further instances of this message will not be shown
2023-06-23T20:05:06.009 INFO:tasks.ceph.mon.c.smithi188.stderr:*** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.009 INFO:tasks.ceph.mon.c.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.010 INFO:tasks.ceph.mon.a.smithi133.stderr:     0> 2023-06-23T20:05:05.972+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.010 INFO:tasks.ceph.mon.a.smithi133.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.010 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:06.010 INFO:tasks.ceph.mon.a.smithi133.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.011 INFO:tasks.ceph.mon.a.smithi133.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 11: main()
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: 13: _start()
2023-06-23T20:05:06.012 INFO:tasks.ceph.mon.a.smithi133.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.013 INFO:tasks.ceph.mon.a.smithi133.stderr:
2023-06-23T20:05:06.072 INFO:tasks.ceph.mon.b.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.073 INFO:tasks.ceph.mon.b.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 11: main()
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.074 INFO:tasks.ceph.mon.b.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.083 INFO:tasks.ceph.mon.c.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.083 INFO:tasks.ceph.mon.c.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.084 INFO:tasks.ceph.mon.c.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.085 INFO:tasks.ceph.mon.c.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.085 INFO:tasks.ceph.mon.c.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.085 INFO:tasks.ceph.mon.c.smithi188.stderr: 11: main()
2023-06-23T20:05:06.085 INFO:tasks.ceph.mon.c.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.085 INFO:tasks.ceph.mon.c.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.100 INFO:tasks.ceph.mon.b.smithi188.stderr:2023-06-23T20:05:06.089+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.101 INFO:tasks.ceph.mon.b.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.102 INFO:tasks.ceph.mon.b.smithi188.stderr: 11: main()
2023-06-23T20:05:06.103 INFO:tasks.ceph.mon.b.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.103 INFO:tasks.ceph.mon.b.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.103 INFO:tasks.ceph.mon.b.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.103 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.112 INFO:tasks.ceph.mon.b.smithi188.stderr:     0> 2023-06-23T20:05:06.089+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.113 INFO:tasks.ceph.mon.b.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.113 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.113 INFO:tasks.ceph.mon.b.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.114 INFO:tasks.ceph.mon.b.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.114 INFO:tasks.ceph.mon.b.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.114 INFO:tasks.ceph.mon.b.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.115 INFO:tasks.ceph.mon.b.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.116 INFO:tasks.ceph.mon.b.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.116 INFO:tasks.ceph.mon.b.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.116 INFO:tasks.ceph.mon.b.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.116 INFO:tasks.ceph.mon.b.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.117 INFO:tasks.ceph.mon.b.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.117 INFO:tasks.ceph.mon.b.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.117 INFO:tasks.ceph.mon.b.smithi188.stderr: 11: main()
2023-06-23T20:05:06.118 INFO:tasks.ceph.mon.b.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.118 INFO:tasks.ceph.mon.b.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.118 INFO:tasks.ceph.mon.b.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.119 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.119 INFO:tasks.ceph.mon.c.smithi188.stderr:2023-06-23T20:05:06.097+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.120 INFO:tasks.ceph.mon.c.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.120 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.120 INFO:tasks.ceph.mon.c.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.121 INFO:tasks.ceph.mon.c.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.121 INFO:tasks.ceph.mon.c.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.121 INFO:tasks.ceph.mon.c.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.121 INFO:tasks.ceph.mon.c.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.122 INFO:tasks.ceph.mon.c.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.122 INFO:tasks.ceph.mon.c.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.122 INFO:tasks.ceph.mon.c.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.123 INFO:tasks.ceph.mon.c.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.123 INFO:tasks.ceph.mon.c.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.123 INFO:tasks.ceph.mon.c.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.124 INFO:tasks.ceph.mon.c.smithi188.stderr: 11: main()
2023-06-23T20:05:06.124 INFO:tasks.ceph.mon.c.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.124 INFO:tasks.ceph.mon.c.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.124 INFO:tasks.ceph.mon.c.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.125 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.126 INFO:tasks.ceph.mon.c.smithi188.stderr:     0> 2023-06-23T20:05:06.097+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.126 INFO:tasks.ceph.mon.c.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.126 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.126 INFO:tasks.ceph.mon.c.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.127 INFO:tasks.ceph.mon.c.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.127 INFO:tasks.ceph.mon.c.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.127 INFO:tasks.ceph.mon.c.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.128 INFO:tasks.ceph.mon.c.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.128 INFO:tasks.ceph.mon.c.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.128 INFO:tasks.ceph.mon.c.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.129 INFO:tasks.ceph.mon.c.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.129 INFO:tasks.ceph.mon.c.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.129 INFO:tasks.ceph.mon.c.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.129 INFO:tasks.ceph.mon.c.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.130 INFO:tasks.ceph.mon.c.smithi188.stderr: 11: main()
2023-06-23T20:05:06.130 INFO:tasks.ceph.mon.c.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.130 INFO:tasks.ceph.mon.c.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.130 INFO:tasks.ceph.mon.c.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.131 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.132 INFO:tasks.ceph.mon.b.smithi188.stderr:     0> 2023-06-23T20:05:06.089+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.132 INFO:tasks.ceph.mon.b.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.132 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.132 INFO:tasks.ceph.mon.b.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.133 INFO:tasks.ceph.mon.b.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.133 INFO:tasks.ceph.mon.b.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.133 INFO:tasks.ceph.mon.b.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.133 INFO:tasks.ceph.mon.b.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.133 INFO:tasks.ceph.mon.b.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.134 INFO:tasks.ceph.mon.b.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.134 INFO:tasks.ceph.mon.b.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.134 INFO:tasks.ceph.mon.b.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.134 INFO:tasks.ceph.mon.b.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.135 INFO:tasks.ceph.mon.b.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.135 INFO:tasks.ceph.mon.b.smithi188.stderr: 11: main()
2023-06-23T20:05:06.135 INFO:tasks.ceph.mon.b.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.135 INFO:tasks.ceph.mon.b.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.135 INFO:tasks.ceph.mon.b.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.136 INFO:tasks.ceph.mon.b.smithi188.stderr:
2023-06-23T20:05:06.139 INFO:tasks.ceph.mon.c.smithi188.stderr:     0> 2023-06-23T20:05:06.097+0000 708d7c0 -1 *** Caught signal (Segmentation fault) **
2023-06-23T20:05:06.139 INFO:tasks.ceph.mon.c.smithi188.stderr: in thread 708d7c0 thread_name:ceph-mon
2023-06-23T20:05:06.139 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.139 INFO:tasks.ceph.mon.c.smithi188.stderr: ceph version 18.1.0-198-g40464acd (40464acdd8735105ba505dacc5bf9acb29b6dcab) reef (rc)
2023-06-23T20:05:06.139 INFO:tasks.ceph.mon.c.smithi188.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2023-06-23T20:05:06.140 INFO:tasks.ceph.mon.c.smithi188.stderr: 2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
2023-06-23T20:05:06.140 INFO:tasks.ceph.mon.c.smithi188.stderr: 3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
2023-06-23T20:05:06.140 INFO:tasks.ceph.mon.c.smithi188.stderr: 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
2023-06-23T20:05:06.140 INFO:tasks.ceph.mon.c.smithi188.stderr: 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
2023-06-23T20:05:06.141 INFO:tasks.ceph.mon.c.smithi188.stderr: 6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
2023-06-23T20:05:06.141 INFO:tasks.ceph.mon.c.smithi188.stderr: 7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109fb5d]
2023-06-23T20:05:06.141 INFO:tasks.ceph.mon.c.smithi188.stderr: 8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109ebb9]
2023-06-23T20:05:06.141 INFO:tasks.ceph.mon.c.smithi188.stderr: 9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf5b800]
2023-06-23T20:05:06.141 INFO:tasks.ceph.mon.c.smithi188.stderr: 10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5d99d]
2023-06-23T20:05:06.142 INFO:tasks.ceph.mon.c.smithi188.stderr: 11: main()
2023-06-23T20:05:06.142 INFO:tasks.ceph.mon.c.smithi188.stderr: 12: __libc_start_main()
2023-06-23T20:05:06.142 INFO:tasks.ceph.mon.c.smithi188.stderr: 13: _start()
2023-06-23T20:05:06.142 INFO:tasks.ceph.mon.c.smithi188.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2023-06-23T20:05:06.143 INFO:tasks.ceph.mon.c.smithi188.stderr:
2023-06-23T20:05:06.690 INFO:tasks.ceph.mon.a.smithi133.stderr:daemon-helper: command crashed with signal 11
2023-06-23T20:05:06.695 INFO:tasks.ceph.mon.b.smithi188.stderr:daemon-helper: command crashed with signal 11
2023-06-23T20:05:06.703 INFO:tasks.ceph.mon.c.smithi188.stderr:daemon-helper: command crashed with signal 11

Traceback is as follows:

2023-06-23T21:44:55.585 INFO:teuthology.orchestra.run.smithi133.stderr:[errno 110] RADOS timed out (error connecting to the cluster)
2023-06-23T21:44:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-06-23T21:44:55.589 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_40464acdd8735105ba505dacc5bf9acb29b6dcab/qa/tasks/ceph.py", line 373, in crush_setup
    mon_remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

Related issues 3 (2 open — 1 closed)

Actions

Copy link

Updated by Milind Changire 10 months ago

reef:
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-23_16:20:29-fs-wip-yuri11-testing-2023-06-19-1232-reef-distro-default-smithi/7313662

Actions

Copy link

Updated by Milind Changire 10 months ago

reef:
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-23_16:20:29-fs-wip-yuri11-testing-2023-06-19-1232-reef-distro-default-smithi/7313675

Actions

Copy link

Updated by Radoslaw Zarzynski 10 months ago

Assignee set to Matan Breizman

Is this reproducible on the branches as well? My intention is figure out whether it's regression in Reef.

Also, isn't the backtrace suggesting an out-of-memory situation?

Actions

Copy link

Updated by Matan Breizman 10 months ago

Subject changed from CommandFailedError: Command failed on smithi133 with status 1: 'sudo ceph --cluster ceph osd crush tunables default' to mon: segfault on rocksdb opening

Changing the tracker subject since the failed command is arbitrary (first one used).

The mons never started as the crash appears early on while initializing rocksdb.

Actions

Copy link

Updated by Matan Breizman 10 months ago

Is this reproducible on the branches as well? My intention is figure out whether it's regression in Reef.

Rescheduling the reported jobs in R and main to check whether it's reproducible.

Also, isn't the backtrace suggesting an out-of-memory situation?

Are you referring to the `allocate_full_cpp_throw_oom` or the fact that the mons weren't able to start at all?

Actions

Copy link

Updated by Matan Breizman 10 months ago

Backport set to reef

I rerun the same jobs 10 times each with the tasks of dbench and fsstress.
Main looks good:
fsstress 10/10 [1]
dbench 9/10 [2] - the single failure is not similar to the one reported, no monitors segfaulted according to the logs.

However in Reef [3]⁴, all 20 jobs failed with all the monitor crashing on rocksdb opening.

This issue appear to relate to Reef (and not to main) I will test Q also and look for related changes in main.

[1] https://pulpito.ceph.com/matan-2023-07-13_12:13:13-fs:verify-main-distro-default-smithi/
[2] https://pulpito.ceph.com/matan-2023-07-13_12:11:45-fs:verify-main-distro-default-smithi/
[3] https://pulpito.ceph.com/matan-2023-07-13_12:15:57-fs:verify-reef-distro-default-smithi/
[4] https://pulpito.ceph.com/matan-2023-07-13_12:15:14-fs:verify-reef-distro-default-smithi/

I have attached few trackers reported recently by Telemetry with a similar tcmalloc backtrace. (Might be related, although most likely these are common tcmalloc memory corruption errors)

Same RocksDB version of: 7.9.2 (both in R and main)

valgrind's first reported error:

<error>
  <unique>0x1</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x55F50A0</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)</fn>
    </frame>
    <frame>
      <ip>0x55F536F</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)</fn>
    </frame>
    <frame>
      <ip>0x55F542F</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)</fn>
    </frame>
    <frame>
      <ip>0x55F8E45</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))</fn>
    </frame>
    <frame>
      <ip>0x5609014</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
      <fn>tcmalloc::allocate_full_cpp_throw_oom(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x109FB5C</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**, bool, bool)</fn>
    </frame>
    <frame>
      <ip>0x109EBB8</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::DB::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**)</fn>
    </frame>
    <frame>
      <ip>0xF5B7FF</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>RocksDBStore::do_open(std::ostream&amp;, bool, bool, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)</fn>
      <dir>./obj-x86_64-linux-gnu/src/kv/./src/kv</dir>
      <file>RocksDBStore.cc</file>
      <line>1193</line>
    </frame>
    <frame>
      <ip>0xC5D99C</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>MonitorDBStore::open(std::ostream&amp;)</fn>
      <dir>./obj-x86_64-linux-gnu/src/./src/mon</dir>
      <file>MonitorDBStore.h</file>
      <line>674</line>
    </frame>
    <frame>
      <ip>0xC379EF</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>main</fn>
      <dir>./obj-x86_64-linux-gnu/src/./src</dir>
      <file>ceph_mon.cc</file>
      <line>639</line>
    </frame>
  </stack>
  <auxwhat>Address 0x20 is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>

Actions

Copy link