Ceph » CephFS

While performing an mds cache drop I had an MDS assert.

Command was:

> ceph daemon mds.`hostname -s` cache drop 10

Crash line was:

0> 2019-07-01 15:25:20.880887 7f5957186700 -1 /builddir/build/BUILD/ceph-12.2.12/src/include/Context.h: In function 'C_GatherBuilderBase<ContextType, GatherType>::~C_GatherBuilderBase() [with ContextType = MDSInternalContextBase; GatherType = MDSGather]' thread 7f5957186700 time 2019-07-01 15:25:20.877570
/builddir/build/BUILD/ceph-12.2.12/src/include/Context.h: 432: FAILED assert(activated)

Backtrace:

(gdb) bt
#0  0x00007f595e9dc49b in raise () from /lib64/libpthread.so.0
#1  0x000055a7faafcb86 in reraise_fatal (signum=6) at /usr/src/debug/ceph-12.2.12/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=6) at /usr/src/debug/ceph-12.2.12/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  0x00007f595d9ea207 in raise () from /lib64/libc.so.6
#5  0x00007f595d9eb8f8 in abort () from /lib64/libc.so.6
#6  0x000055a7fab3e6d4 in ceph::__ceph_assert_fail (assertion=assertion@entry=0x55a7faf20f90 "activated", 
    file=file@entry=0x55a7faf1ed98 "/builddir/build/BUILD/ceph-12.2.12/src/include/Context.h", line=line@entry=432, 
    func=func@entry=0x55a7faf22540 <C_GatherBuilderBase<MDSInternalContextBase, MDSGather>::~C_GatherBuilderBase()::__PRETTY_FUNCTION__> "C_GatherBuilderBase<ContextType, GatherType>::~C_GatherBuilderBase() [with ContextType = MDSInternalContextBase; GatherType = MDSGather]") at /usr/src/debug/ceph-12.2.12/src/common/assert.cc:66
#7  0x000055a7fa8196d2 in C_GatherBuilderBase<MDSInternalContextBase, MDSGather>::~C_GatherBuilderBase (this=<optimized out>, __in_chrg=<optimized out>)
    at /usr/src/debug/ceph-12.2.12/src/include/Context.h:432
#8  0x000055a7fa82092b in C_Drop_Cache::recall_client_state (this=0x55a9106c7680) at /usr/src/debug/ceph-12.2.12/src/mds/MDSRank.cc:343
#9  0x000055a7fa7efe1a in operator() (a0=<optimized out>, this=<optimized out>) at /usr/src/debug/ceph-12.2.12/build/boost/include/boost/function/function_template.hpp:760
#10 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/include/Context.h:493
#11 0x000055a7fa7ecfe9 in Context::complete (this=0x55a8f031db60, r=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/include/Context.h:70
#12 0x000055a7fab3ae04 in SafeTimer::timer_thread (this=0x55a80596a078) at /usr/src/debug/ceph-12.2.12/src/common/Timer.cc:97
#13 0x000055a7fab3c82d in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/common/Timer.cc:30
#14 0x00007f595e9d4dd5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f595dab1ead in clone () from /lib64/libc.so.6

Some gdb:

(gdb) up
#8  0x000055a7fa82092b in C_Drop_Cache::recall_client_state (this=0x55a9106c7680) at /usr/src/debug/ceph-12.2.12/src/mds/MDSRank.cc:343
343            delete gather;
(gdb) p gather
$1 = (MDSGatherBuilder *) 0x55a8a8587820
(gdb) p *gather
$2 = {cct = 0x55a8057081c0, c_gather = 0x55a811b2ad80, finisher = 0x0, activated = false}
(gdb)