Project

General

Profile

Bug #47243

bluefs _allocate failed then assert

Added by chunsong feng 18 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
leveldb
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

-4> 2020-09-01T21:55:35.451+0800 ffffb5f50500 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
-3> 2020-09-01T21:55:35.511+0800 ffffb5f50500 1 bluefs _allocate failed to allocate 0x400000 on bdev 1, free 0x165ef7e000; fallback to bdev 2
-2> 2020-09-01T21:55:35.511+0800 ffffb5f50500 1 bluefs _allocate unable to allocate 0x400000 on bdev 2, free 0xffffffffffffffff
-1> 2020-09-01T21:55:35.521+0800 ffffb5f50500 -1 /home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)' thread ffffb5f50500 time 2020-09-01T21:55:35.527170+0800
/home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: 2447: FAILED ceph_assert(r == 0)

ceph version 16.0.0-4421-gcfeea548ad (cfeea548adc3428b5d423ac43556eaf8f37b6cbc) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0xaaaacafca9a4]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xaaaacafcab8c]
3: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0x182c) [0xaaaacb5ca724]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x7c) [0xaaaacb5caa9c]
5: (BlueRocksWritableFile::Flush()+0x78) [0xaaaacb5e80c0]
6: (rocksdb::LegacyWritableFileWrapper::Flush(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaaacba20014]
7: (rocksdb::WritableFileWriter::Flush()+0x1c0) [0xaaaacbb0edd0]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaaacbb0f6d8]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaaacbc0d958]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaacbc0e0a4]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaacbc0e640]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaacbc0e6bc]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaaacbc1139c]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaaacbbc6498]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaacba586e4]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaaacba59fe0]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaacba5b74c]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaaacba56604]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaaacba576dc]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcec) [0xaaaacb9dfd3c]
21: (BlueStore::_open_db(bool, bool, bool)+0x78c) [0xaaaacb4c7fb4]
22: (BlueStore::_open_db_and_around(bool)+0xc8) [0xaaaacb4dc6d8]
23: (BlueStore::_mount(bool, bool)+0x7f8) [0xaaaacb52a7d0]
24: (OSD::init()+0x20c) [0xaaaacb092bc4]
25: (main()+0x3e5c) [0xaaaacafd19d4]
26: (__libc_start_main()+0xe4) [0xffffb6370be4]
27: (()+0x5391b8) [0xaaaacb0191b8]

0> 2020-09-01T21:55:35.531+0800 ffffb5f50500 -1 ** Caught signal (Aborted) *
in thread ffffb5f50500 thread_name:ceph-osd

ceph version 16.0.0-4421-gcfeea548ad (cfeea548adc3428b5d423ac43556eaf8f37b6cbc) pacific (dev)
1: (_kernel_rt_sigreturn()+0) [0xffffb70e066c]
2: (gsignal()+0xac) [0xffffb6382c3c]
3: (abort()+0x110) [0xffffb63707a8]
4: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x1a4) [0xaaaacafca9f4]
5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xaaaacafcab8c]
6: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0x182c) [0xaaaacb5ca724]
7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x7c) [0xaaaacb5caa9c]
8: (BlueRocksWritableFile::Flush()+0x78) [0xaaaacb5e80c0]
9: (rocksdb::LegacyWritableFileWrapper::Flush(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaaacba20014]
10: (rocksdb::WritableFileWriter::Flush()+0x1c0) [0xaaaacbb0edd0]
11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaaacbb0f6d8]
12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaaacbc0d958]
13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaacbc0e0a4]
14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaacbc0e640]
15: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaacbc0e6bc]
16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaaacbc1139c]
17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaaacbbc6498]
18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaacba586e4]
19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaaacba59fe0]
20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaacba5b74c]
21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaaacba56604]
22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaaacba576dc]
23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcec) [0xaaaacb9dfd3c]
24: (BlueStore::_open_db(bool, bool, bool)+0x78c) [0xaaaacb4c7fb4]
25: (BlueStore::_open_db_and_around(bool)+0xc8) [0xaaaacb4dc6d8]
26: (BlueStore::_mount(bool, bool)+0x7f8) [0xaaaacb52a7d0]
27: (OSD::init()+0x20c) [0xaaaacb092bc4]
28: (main()+0x3e5c) [0xaaaacafd19d4]
29: (__libc_start_main()+0xe4) [0xffffb6370be4]
30: (()+0x5391b8) [0xaaaacb0191b8]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

bluefs_assert.txt View (401 KB) chunsong feng, 09/02/2020 12:43 AM

Also available in: Atom PDF