Project

General

Profile

Fix #54299

osd error restart

Added by duans song about 2 years ago. Updated over 1 year ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

debug -83> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
debug -82> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
debug -81> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
debug -80> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.bottommost_compression_opts.enabled: false
debug -79> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.window_bits: -14
debug -78> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.level: 32767
debug -77> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.strategy: 0
debug -76> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
debug -75> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
debug -74> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.compression_opts.enabled: false
debug -73> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
debug -72> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
debug -71> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.level0_stop_writes_trigger: 36
debug -70> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.target_file_size_base: 67108864
debug -69> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.target_file_size_multiplier: 1
debug -68> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_base: 268435456
debug -67> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
debug -66> 2022-02-16T13:04:03.711+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
debug -65> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl0: 1
debug -64> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl1: 1
debug -63> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl2: 1
debug -62> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl3: 1
debug -61> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl4: 1
debug -60> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl5: 1
debug -59> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl6: 1
debug -58> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
debug -57> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_compaction_bytes: 1677721600
debug -56> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.arena_block_size: 33554432
debug -55> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
debug -54> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
debug -53> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
debug -52> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.disable_auto_compactions: 0
debug -51> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
debug -50> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
debug -49> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
debug -48> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
debug -47> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
debug -46> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
debug -45> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
debug -44> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
debug -43> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
debug -42> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
debug -41> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.table_properties_collectors:
debug -40> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.inplace_update_support: 0
debug -39> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.inplace_update_num_locks: 10000
debug -38> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
debug -37> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.memtable_whole_key_filtering: 0
debug -36> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.memtable_huge_page_size: 0
debug -35> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.bloom_locality: 0
debug -34> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.max_successive_merges: 0
debug -33> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.optimize_filters_for_hits: 0
debug -32> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.paranoid_file_checks: 0
debug -31> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.force_consistency_checks: 0
debug -30> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.report_bg_io_stats: 0
debug -29> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.ttl: 2592000
debug -28> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: Options.periodic_compaction_seconds: 0
debug -27> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: [column_family.cc:555] (skipping printing options)

debug -26> 2022-02-16T13:04:03.712+0000 7f35fee9d080 4 rocksdb: [column_family.cc:555] (skipping printing options)

debug -25> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-044787 succeeded,manifest_file_number is 44787, next_file_number is 45179, last_sequence is 974768953, log_number is 45176,prev_log_number is 0,max_column_family is 11,min_log_number_to_keep is 0

debug -24> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 45157

debug -23> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1), log number is 44777

debug -22> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2), log number is 44777

debug -21> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3), log number is 44777

debug -20> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4), log number is 44777

debug -19> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5), log number is 44777

debug -18> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6), log number is 44777

debug -17> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7), log number is 45162

debug -16> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8), log number is 45157

debug -15> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9), log number is 45162

debug -14> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10), log number is 45176

debug -13> 2022-02-16T13:04:03.731+0000 7f35fee9d080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11), log number is 45157

debug -12> 2022-02-16T13:04:03.732+0000 7f35fee9d080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1645016643732799, "job": 1, "event": "recovery_started", "log_files": [45157, 45162, 45168, 45171, 45173, 45176, 45178]}
debug -11> 2022-02-16T13:04:03.732+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45157 mode 2
debug -10> 2022-02-16T13:04:03.801+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45162 mode 2
debug -9> 2022-02-16T13:04:04.018+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45168 mode 2
debug -8> 2022-02-16T13:04:04.931+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45171 mode 2
debug -7> 2022-02-16T13:04:05.847+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45173 mode 2
debug -6> 2022-02-16T13:04:06.798+0000 7f35fee9d080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #45176 mode 2
debug -5> 2022-02-16T13:04:07.816+0000 7f35fee9d080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
debug -4> 2022-02-16T13:04:07.980+0000 7f35fee9d080 1 bluefs _allocate unable to allocate 0x400000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x8000000000, block size 0x1000, free 0x3909b57000, fragmentation 0.0523383, allocated 0x0
debug -3> 2022-02-16T13:04:07.980+0000 7f35fee9d080 -1 bluefs _allocate allocation failed, needed 0x3fd8ae
debug -2> 2022-02-16T13:04:07.980+0000 7f35fee9d080 -1 bluefs _flush_range allocated: 0x220000 offset: 0x21d430 length: 0x40047e
debug -1> 2022-02-16T13:04:07.990+0000 7f35fee9d080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f35fee9d080 time 2022-02-16T13:04:07.981279+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/os/bluestore/BlueFS.cc: 2729: ceph_abort_msg("bluefs enospc")

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x55b9ff12d7a4]
2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1614) [0x55b9ff81e914]
3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55b9ff81ec80]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x55b9ff838362]
5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x55b9ff84a67b]
6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55b9ffcdbb2f]
7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x55b9ffded88a]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x55b9ffdeece0]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x55b9fff0a496]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x55b9fff0addc]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x55b9fff0b4fc]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x55b9fff0b58d]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x55b9fff0e9f8]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x55b9ffeb9455]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55b9ffd1e475]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1491) [0x55b9ffd20411]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55b9ffd21f08]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x59d) [0x55b9ffd1bc2d]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x15) [0x55b9ffd1cfc5]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x55b9ffc94c91]
21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x55b9ff71aa98]
22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x55b9ff784347]
23: (BlueStore::_mount()+0x204) [0x55b9ff787204]
24: (OSD::init()+0x380) [0x55b9ff260e40]
25: main()
26: __libc_start_main()
27: _start()

debug 0> 2022-02-16T13:04:08.000+0000 7f35fee9d080 -1 ** Caught signal (Aborted) *
in thread 7f35fee9d080 thread_name:ceph-osd

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f35fcc04b20]
2: gsignal()
3: abort()
4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55b9ff12d875]
5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1614) [0x55b9ff81e914]
6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55b9ff81ec80]
7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x55b9ff838362]
8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x55b9ff84a67b]
9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55b9ffcdbb2f]
10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x55b9ffded88a]
11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x55b9ffdeece0]
12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x55b9fff0a496]
13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x55b9fff0addc]
14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x55b9fff0b4fc]
15: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x55b9fff0b58d]
16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x55b9fff0e9f8]
17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x55b9ffeb9455]
18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55b9ffd1e475]
19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1491) [0x55b9ffd20411]
20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55b9ffd21f08]
21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x59d) [0x55b9ffd1bc2d]
22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x15) [0x55b9ffd1cfc5]
23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x55b9ffc94c91]
24: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x55b9ff71aa98]
25: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x55b9ff784347]
26: (BlueStore::_mount()+0x204) [0x55b9ff787204]
27: (OSD::init()+0x380) [0x55b9ff260e40]
28: main()
29: __libc_start_main()
30: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
2/-2 (syslog threshold)
99/99 (stderr threshold)
--
pthread ID / name mapping for recent threads ---
139869758236416 / admin_socket
139869766629120 / msgr-worker-2
139869775021824 / msgr-worker-1
139869783414528 / msgr-worker-0
139869886730368 / ceph-osd
max_recent 10000
max_new 10000
log_file /var/lib/ceph/crash/2022-02-16T13:04:08.000688Z_d29e695a-b962-4ec0-8b86-28a1176bf244/log
--- end dump of recent events ---
reraise_fatal: default handler for signal 6 didn't terminate the process?


Related issues

Related to bluestore - Bug #53466: OSD is unable to allocate free space for BlueFS Resolved

History

#1 Updated by Igor Fedotov about 2 years ago

  • Project changed from Ceph to bluestore
  • Category deleted (OSD)

Could you please share the output for ceph-bluestore-tool's free-dump command?

#2 Updated by duans song about 2 years ago

Igor Fedotov wrote:

Could you please share the output for ceph-bluestore-tool's free-dump command?

What command do I need to execute?

#3 Updated by Igor Fedotov about 2 years ago

duans song wrote:

Igor Fedotov wrote:

Could you please share the output for ceph-bluestore-tool's free-dump command?

What command do I need to execute?

ceph-bluestore-tool --path <path-to-osd-dir> --command free-dump

#4 Updated by duans song about 2 years ago

Igor Fedotov wrote:

duans song wrote:

Igor Fedotov wrote:

Could you please share the output for ceph-bluestore-tool's free-dump command?

What command do I need to execute?

ceph-bluestore-tool --path <path-to-osd-dir> --command free-dump

ok, but we use rook to deploy in kubernetes, i can't execute in osd node, i use toolbox to warn me the path is wrong

#5 Updated by Igor Fedotov about 2 years ago

Well, unfortunately I'm not an expert in using rook deployments hence unable to provide any help on how to run the tool in that env. But actually I think my last comment from https://tracker.ceph.com/issues/53466 is applicable for your case. Please take a look.
And yeah, I would advise to start using separate volume for DB before we have a good solution for the issue...

#6 Updated by Igor Fedotov about 2 years ago

  • Related to Bug #53466: OSD is unable to allocate free space for BlueFS added

#7 Updated by duans song about 2 years ago

Igor Fedotov wrote:

Well, unfortunately I'm not an expert in using rook deployments hence unable to provide any help on how to run the tool in that env. But actually I think my last comment from https://tracker.ceph.com/issues/53466 is applicable for your case. Please take a look.
And yeah, I would advise to start using separate volume for DB before we have a good solution for the issue...

--
I think the osd container oom has been restarted too many times, and the process status has not been updated, because this problem has appeared before and suddenly recovered. Now the problem again, no recovery

#8 Updated by Ilya Dryomov over 1 year ago

  • Target version deleted (v16.2.8)

#9 Updated by Igor Fedotov over 1 year ago

  • Status changed from New to Need More Info

Also available in: Atom PDF