Project

General

Profile

Actions

Bug #24906

closed

fio with bluestore crushed

Added by Honggang Yang almost 6 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

(gdb) bt
#0  0x00007f62fff954ab in raise () from /lib64/libpthread.so.0
#1  0x00007f62f6e2c684 in reraise_fatal (signum=11) at /home/yhg/work/pufa/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11) at /home/yhg/work/pufa/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  0x00007f62ffa3fd86 in malloc_usable_size () from /lib64/libc.so.6
#5  0x00007f62f6e9e02f in rocksdb::BlockBasedTable::NewIndexIterator (this=this@entry=0x1e37da0, read_options=..., input_iter=input_iter@entry=
    0x0, index_entry=index_entry@entry=0x0) at /home/yhg/work/pufa/src/rocksdb/table/block_based_table_reader.cc:1198
#6  0x00007f62f6ea2895 in rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool, bool, int) (
    ioptions=..., env_options=..., table_options=..., internal_comparator=..., file=<optimized out>, file_size=1455, 
    table_reader=0x7ffd05669ce8, prefetch_index_and_filter_in_cache=true, skip_filters=false, level=-1)
    at /home/yhg/work/pufa/src/rocksdb/table/block_based_table_reader.cc:726
#7  0x00007f62f6e98b8e in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const (this=<optimized out>, table_reader_options=..., file=<optimized out>, file_size=<optimized out>, 
    table_reader=<optimized out>, prefetch_index_and_filter_in_cache=true)
    at /home/yhg/work/pufa/src/rocksdb/table/block_based_table_factory.cc:59
#8  0x00007f62f6f6cead in rocksdb::TableCache::GetTableReader (this=this@entry=0x1e37ee0, env_options=..., internal_comparator=..., fd=..., 
    sequential_mode=sequential_mode@entry=false, readahead=0, record_read_stats=true, file_read_hist=0x1e9a800, table_reader=0x7ffd05669ce8, 
    skip_filters=false, level=-1, prefetch_index_and_filter_in_cache=true) at /home/yhg/work/pufa/src/rocksdb/db/table_cache.cc:111
#9  0x00007f62f6f6d488 in rocksdb::TableCache::FindTable (this=this@entry=0x1e37ee0, env_options=..., internal_comparator=..., fd=..., 
    handle=handle@entry=0x7ffd05669dd8, no_io=false, record_read_stats=true, file_read_hist=0x1e9a800, skip_filters=false, level=-1, 
    prefetch_index_and_filter_in_cache=true) at /home/yhg/work/pufa/src/rocksdb/db/table_cache.cc:148
#10 0x00007f62f6f6dbd3 in rocksdb::TableCache::NewIterator (this=this@entry=0x1e37ee0, options=..., env_options=..., icomparator=..., fd=..., 
    range_del_agg=range_del_agg@entry=0x0, table_reader_ptr=0x0, file_read_hist=0x1e9a800, for_compaction=false, arena=0x0, skip_filters=false, 
    level=-1) at /home/yhg/work/pufa/src/rocksdb/db/table_cache.cc:213
#11 0x00007f62f6eeff73 in rocksdb::BuildTable (dbname=..., env=0x1e38120, ioptions=..., mutable_cf_options=..., env_options=..., 
    table_cache=0x1e37ee0, iter=0x7ffd0566aa38, range_del_iter=..., meta=0x7ffd0566a7c0, internal_comparator=..., 
    int_tbl_prop_collector_factories=0x1f16540, column_family_id=0, column_family_name=..., snapshots=..., 
    earliest_write_conflict_snapshot=72057594037927935, compression=<incomplete type>, compression_opts=..., paranoid_file_checks=false, 
    internal_stats=0x1ca8c80, reason=rocksdb::kRecovery, event_logger=0x1efd670, job_id=1, io_priority=rocksdb::Env::IO_HIGH, 
    table_properties=0x0, level=-1) at /home/yhg/work/pufa/src/rocksdb/db/builder.cc:199
#12 0x00007f62f6e329f7 in rocksdb::DBImpl::WriteLevel0TableForRecovery (this=this@entry=0x1efcc00, job_id=job_id@entry=1, 
    cfd=cfd@entry=0x1f16500, mem=0x1efd800, edit=edit@entry=0x1c90f30) at /home/yhg/work/pufa/src/rocksdb/db/db_impl_open.cc:831
#13 0x00007f62f6e34a9b in rocksdb::DBImpl::RecoverLogFiles (this=this@entry=0x1efcc00, log_numbers=..., 
    next_sequence=next_sequence@entry=0x7ffd0566b6b0, read_only=read_only@entry=false) at /home/yhg/work/pufa/src/rocksdb/db/db_impl_open.cc:747
#14 0x00007f62f6e3572f in rocksdb::DBImpl::Recover (this=this@entry=0x1efcc00, column_families=..., read_only=read_only@entry=false, 
    error_if_log_file_exist=error_if_log_file_exist@entry=false, error_if_data_exists_in_logs=error_if_data_exists_in_logs@entry=false)
    at /home/yhg/work/pufa/src/rocksdb/db/db_impl_open.cc:391
#15 0x00007f62f6e36b08 in rocksdb::DB::Open (db_options=..., dbname=..., column_families=..., handles=handles@entry=0x7ffd0566bf40, 
    dbptr=dbptr@entry=0x1c6a840) at /home/yhg/work/pufa/src/rocksdb/db/db_impl_open.cc:936
#16 0x00007f62f6e38199 in rocksdb::DB::Open (options=..., dbname=..., dbptr=dbptr@entry=0x1c6a840)
    at /home/yhg/work/pufa/src/rocksdb/db/db_impl_open.cc:881
#17 0x00007f62f6d70d18 in RocksDBStore::do_open (this=0x1c6a800, out=..., create_if_missing=<optimized out>)
    at /home/yhg/work/pufa/src/kv/RocksDBStore.cc:368
#18 0x00007f62f6d06153 in BlueStore::_open_db (this=this@entry=0x1efc000, create=create@entry=false)
    at /home/yhg/work/pufa/src/os/bluestore/BlueStore.cc:4802
#19 0x00007f62f6d2b27f in BlueStore::_fsck (this=this@entry=0x1efc000, deep=false, repair=repair@entry=false)
    at /home/yhg/work/pufa/src/os/bluestore/BlueStore.cc:5662
#20 0x00007f62f6d35076 in fsck (deep=<optimized out>, this=0x1efc000) at /home/yhg/work/pufa/src/os/bluestore/BlueStore.h:2172
#21 BlueStore::mkfs (this=0x1efc000) at /home/yhg/work/pufa/src/os/bluestore/BlueStore.cc:5369
#22 0x00007f62f6bd3826 in Engine (this=0x7f62f729fda0 <(anonymous namespace)::Engine::get_instance(thread_data*)::engine>, td=0x7f62d20bc000)
    at /home/yhg/work/pufa/src/test/fio/fio_ceph_objectstore.cc:147
#23 get_instance (td=0x7f62d20bc000) at /home/yhg/work/pufa/src/test/fio/fio_ceph_objectstore.cc:77
#24 (anonymous namespace)::fio_ceph_os_setup (td=0x7f62d20bc000) at /home/yhg/work/pufa/src/test/fio/fio_ceph_objectstore.cc:283
#25 0x0000000000438944 in setup_files (td=td@entry=0x7f62d20bc000) at filesetup.c:921
#26 0x000000000046b7c7 in run_threads (sk_out=sk_out@entry=0x0) at backend.c:2233
#27 0x000000000046c182 in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2479
#28 0x000000000040deb1 in main (argc=2, argv=0x7ffd05675728, envp=<optimized out>) at fio.c:60

my ceph conf file's content is:

# cat ceph-bluestore.conf 
# example configuration file for ceph-bluestore.fio

[global]
    debug bluestore = 0/0
    debug bluefs = 0/0
    debug bdev = 0/0
    debug rocksdb = 0/0
    # spread objects over 8 collections
    osd pool default pg num = 8
    # increasing shards can help when scaling number of collections
    osd op num shards = 5

[osd]
    osd objectstore = bluestore

        bluestore block path = /dev/sdi1
        bluestore block db path = /dev/sdk1
        bluestore block wal path = /dev/sdl1

    # use directory= option from fio job file
    osd data = ${fio_dir}

    # log inside fio_dir
    log file = ${fio_dir}/log
Actions #1

Updated by Minghao Cong over 5 years ago

I also met this bug on master branch.

Have you solved it?

0.0

Actions #2

Updated by Honggang Yang over 5 years ago

Minghao Cong wrote:

I also met this bug on master branch.

Have you solved it?

0.0

not yet

Actions #3

Updated by Sage Weil over 5 years ago

Is this still broken?

Actions #4

Updated by Sage Weil over 5 years ago

  • Status changed from New to Need More Info
Actions #5

Updated by Adam Kupczyk over 5 years ago

I have encountered similar problem when fio was linked with libc's malloc.
I solved it by forcibly adding tcmalloc to fio compilation:
LIBS="-ltcmalloc" ./configure
LIBS="-ltcmalloc" make

I assumed that internals of memory management don't much, thus the problem.

Actions #6

Updated by Neha Ojha about 5 years ago

  • Status changed from Need More Info to Closed

Closing this for now based on Adam's theory. Please feel free to reopen if the issue persists.

Actions

Also available in: Atom PDF