Ceph » bluestore

Local env:
[ RUN ] ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2
also did hang.

1. Subtest:
---------------------- 13 / 16 ----------------------
bluestore_min_alloc_size = 65536
max_write = 65536
max_size = 1048576
alignment = 512
bluestore_max_blob_size = 262144
bluestore_compression_mode = none
bluestore_csum_type = none
bluestore_default_buffered_read = true
bluestore_default_buffered_write = true
bluestore_sync_submit_transaction = true

2. Waiting in
#0 0x00007ffff7bbd44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff4332860 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2 0x0000555555ab47a0 in BlueStore::OpSequencer::flush (this=0x55555d6d9a20) at ../src/os/bluestore/BlueStore.h:2136
#3 BlueStore::Collection::flush (this=0x5555569ab4a0) at ../src/os/bluestore/BlueStore.cc:4731
#4 BlueStore::collection_list (this=0x5555721c3000, c_=..., start=..., end=..., max=100, ls=0x7fffffffbc00,
pnext=0x7fffffffbc80) at ../src/os/bluestore/BlueStore.cc:12523
#5 0x0000555555997043 in SyntheticWorkloadState::scan (this=0x7fffffffc010) at ../src/common/hobject.h:466
#6 0x000055555591ca05 in StoreTest::doSyntheticTest (this=<optimized out>, initial_object_count=1000,
num_ops=10000, max_obj=<optimized out>, max_wr=<optimized out>, align=<optimized out>)
at ../src/test/objectstore/store_test.cc:5136

3. Content of class OpSequencer.q (type q_list_t).
(gdb) define list_trans

set $pos = $arg0
set $count = $arg1
while ($count > 0)

>print ((TransContext)$pos).state
>set $pos = (char*)((TransContext)$pos).sequencer_item.next_ - 0x18
>set $count = $count - 1
>end

end

(gdb) print q
$11 = {<boost::intrusive::list_impl<boost::intrusive::mhtraits<BlueStore::TransContext, boost::intrusive::list_member_hook<>, &BlueStore::TransContext::sequencer_item>, long unsigned int, true, void>> = {
static constant_time_size = true, static stateful_value_traits = <optimized out>,
static has_container_from_iterator = <optimized out>, static safemode_or_autounlink = true,
data_ = {<boost::intrusive::mhtraits<BlueStore::TransContext, boost::intrusive::list_member_hook<>, &BlueStore::TransContext::sequencer_item>> = {static link_mode = boost::intrusive::safe_link},
root_plus_size_ = {<boost::intrusive::detail::size_holder<true, unsigned long, void>> = {
static constant_time_size = <optimized out>, size_ = 22},
m_header = {<boost::intrusive::list_node<void*>> = {next_ = 0x555558326f18,
prev_ = 0x5555eface918}, <No data fields>}}}}, <No data fields>}
(gdb) list_trans 0x555558326f00 22
$12 = BlueStore::TransContext::STATE_DEFERRED_CLEANUP
$13 = BlueStore::TransContext::STATE_DONE
$14 = BlueStore::TransContext::STATE_DONE
$15 = BlueStore::TransContext::STATE_DONE
$16 = BlueStore::TransContext::STATE_DONE
$17 = BlueStore::TransContext::STATE_DONE
$18 = BlueStore::TransContext::STATE_DONE
$19 = BlueStore::TransContext::STATE_DONE
$20 = BlueStore::TransContext::STATE_DONE
$21 = BlueStore::TransContext::STATE_DONE
$22 = BlueStore::TransContext::STATE_DONE
$23 = BlueStore::TransContext::STATE_DONE
$24 = BlueStore::TransContext::STATE_DONE
$25 = BlueStore::TransContext::STATE_DONE
$26 = BlueStore::TransContext::STATE_DONE
$27 = BlueStore::TransContext::STATE_DONE
$28 = BlueStore::TransContext::STATE_DONE
$29 = BlueStore::TransContext::STATE_DONE
$30 = BlueStore::TransContext::STATE_DONE
$31 = BlueStore::TransContext::STATE_DONE
$32 = BlueStore::TransContext::STATE_DONE
$33 = BlueStore::TransContext::STATE_DONE

4. kv_sync_thread sleeps:
#0 0x00007ffff7bbd44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff4332860 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2 0x0000555555b24075 in BlueStore::_kv_sync_thread (this=0x5555721c3000) at ../src/os/bluestore/BlueStore.cc:14058
#3 0x0000555555b567fd in BlueStore::KVSyncThread::entry (this=<optimized out>)
at ../src/os/bluestore/BlueStore.h:2199

5. queues @ kv_sync_thread:
(gdb) print deferred_done_queue
$42 = std::deque with 1 element = {0x55555730b680}
(gdb) print deferred_stable_queue
$43 = std::deque with 0 elements
(gdb) print kv_queue
$44 = std::deque with 0 elements

So far, unable to create scenario when such hangup can happen