Ceph » mgr

Description of problem¶

When running the rbd perf image iostat command with enabled rbd metrics in the prometheus exporter ceph config set mgr mgr/prometheus/rbd_stats_pools testpool the active mgr will crash.

Environment¶

• ceph version string: 15.2.14
• Platform (OS/distro/release): CentOS Stream 8

How reproducible¶

Enable the prometheus exporter and the prometheus rbd metrics on an rbd pool with above command. Run rbd perf image iostat --pool {testpool} and watch the mgr crash.

Crash backtrace¶

(gdb) bt
#0  0x00007fe0671af9bf in raise () from /lib64/libpthread.so.0
#1  0x000055b4e5a730c3 in reraise_fatal (signum=6)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/global/signal_handler.cc:326
#2  handle_fatal_signal (signum=6)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/global/signal_handler.cc:326
#3  <signal handler called>
#4  0x00007fe065bf937f in raise () from /lib64/libc.so.6
#5  0x00007fe065be3db5 in abort () from /lib64/libc.so.6
#6  0x000055b4e5854908 in std::__replacement_assert (
    __file=__file@entry=0x55b4e5ab221c "/usr/include/c++/8/optional", 
    __line=__line@entry=910, 
    __function=__function@entry=0x55b4e5abf480 <std::_Optional_base<OSDPerfMetricLimit, true, true>::_M_get() const::__PRETTY_FUNCTION__> "constexpr const _Tp& std::_Optional_base<_Tp, true, true>::_M_get() const [with _Tp = OSDPerfMetricLimit]", 
    __condition=__condition@entry=0x55b4e5ab2206 "this->_M_is_engaged()")
    at /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h:2397
#7  0x000055b4e58fd1ef in std::_Optional_base<OSDPerfMetricLimit, true, true>::_M_get (
    this=<optimized out>) at /usr/include/c++/8/bits/stl_function.h:1131
#8  std::optional<OSDPerfMetricLimit>::operator*() const & (this=<optimized out>)
    at /usr/include/c++/8/optional:1190
#9  MetricCollector<OSDPerfMetricQuery, OSDPerfMetricLimit, std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, OSDPerfMetricReport>::get_queries (this=this@entry=0x55b4eac27488)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/mgr/MetricCollector.h:47
#10 0x000055b4e58cfdb9 in DaemonServer::_send_configure (this=0x55b4eac271e8, c=...)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/mgr/DaemonServer.cc:2905
#11 0x000055b4e58d203d in DaemonServer::handle_open (this=0x55b4eac271e8, m=...)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/common/RefCountedObj.h:56
#12 0x000055b4e58ec1e7 in DaemonServer::ms_dispatch2 (this=0x55b4eac271e8, m=...)
    at /usr/src/debug/ceph-15.2.14-1.el8.cern.x86_64/src/common/RefCountedObj.h:56
#13 0x00007fe068cea93a in DispatchQueue::entry() () from /usr/lib64/ceph/libceph-common.so.2
#14 0x00007fe068d8d981 in DispatchQueue::DispatchThread::entry() ()
   from /usr/lib64/ceph/libceph-common.so.2
#15 0x00007fe0671a514a in start_thread () from /lib64/libpthread.so.0
#16 0x00007fe065cbedc3 in clone () from /lib64/libc.so.6