Bug #47243: bluefs _allocate failed then assert - bluestore - Ceph

Actions

Copy link

Bug #47243

closed

bluefs _allocate failed then assert

Added by chunsong feng over 3 years ago. Updated over 2 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

Ceph - v16.0.0

% Done:

Source:

Community (dev)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

-4> 2020-09-01T21:55:35.451+0800 ffffb5f50500 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
-3> 2020-09-01T21:55:35.511+0800 ffffb5f50500 1 bluefs _allocate failed to allocate 0x400000 on bdev 1, free 0x165ef7e000; fallback to bdev 2
-2> 2020-09-01T21:55:35.511+0800 ffffb5f50500 1 bluefs _allocate unable to allocate 0x400000 on bdev 2, free 0xffffffffffffffff
-1> 2020-09-01T21:55:35.521+0800 ffffb5f50500 -1 /home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)' thread ffffb5f50500 time 2020-09-01T21:55:35.527170+0800
/home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: 2447: FAILED ceph_assert(r == 0)

ceph version 16.0.0-4421-gcfeea548ad (cfeea548adc3428b5d423ac43556eaf8f37b6cbc) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0xaaaacafca9a4]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xaaaacafcab8c]
3: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0x182c) [0xaaaacb5ca724]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x7c) [0xaaaacb5caa9c]
5: (BlueRocksWritableFile::Flush()+0x78) [0xaaaacb5e80c0]
6: (rocksdb::LegacyWritableFileWrapper::Flush(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaaacba20014]
7: (rocksdb::WritableFileWriter::Flush()+0x1c0) [0xaaaacbb0edd0]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaaacbb0f6d8]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaaacbc0d958]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaacbc0e0a4]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaacbc0e640]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaacbc0e6bc]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaaacbc1139c]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaaacbbc6498]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaacba586e4]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaaacba59fe0]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaacba5b74c]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaaacba56604]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaaacba576dc]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcec) [0xaaaacb9dfd3c]
21: (BlueStore::_open_db(bool, bool, bool)+0x78c) [0xaaaacb4c7fb4]
22: (BlueStore::_open_db_and_around(bool)+0xc8) [0xaaaacb4dc6d8]
23: (BlueStore::_mount(bool, bool)+0x7f8) [0xaaaacb52a7d0]
24: (OSD::init()+0x20c) [0xaaaacb092bc4]
25: (main()+0x3e5c) [0xaaaacafd19d4]
26: (__libc_start_main()+0xe4) [0xffffb6370be4]
27: (()+0x5391b8) [0xaaaacb0191b8]

0> 2020-09-01T21:55:35.531+0800 ffffb5f50500 -1 ** Caught signal (Aborted) *
 in thread ffffb5f50500 thread_name:ceph-osd

ceph version 16.0.0-4421-gcfeea548ad (cfeea548adc3428b5d423ac43556eaf8f37b6cbc) pacific (dev)
1: (_kernel_rt_sigreturn()+0) [0xffffb70e066c]
2: (gsignal()+0xac) [0xffffb6382c3c]
3: (abort()+0x110) [0xffffb63707a8]
4: (ceph::_ceph_assert_fail(char const*, char const*, int, char const*)+0x1a4) [0xaaaacafca9f4]
5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xaaaacafcab8c]
6: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0x182c) [0xaaaacb5ca724]
7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x7c) [0xaaaacb5caa9c]
8: (BlueRocksWritableFile::Flush()+0x78) [0xaaaacb5e80c0]
9: (rocksdb::LegacyWritableFileWrapper::Flush(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaaacba20014]
10: (rocksdb::WritableFileWriter::Flush()+0x1c0) [0xaaaacbb0edd0]
11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaaacbb0f6d8]
12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaaacbc0d958]
13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaacbc0e0a4]
14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaacbc0e640]
15: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaacbc0e6bc]
16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaaacbc1139c]
17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaaacbbc6498]
18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaacba586e4]
19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaaacba59fe0]
20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaacba5b74c]
21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaaacba56604]
22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaaacba576dc]
23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcec) [0xaaaacb9dfd3c]
24: (BlueStore::_open_db(bool, bool, bool)+0x78c) [0xaaaacb4c7fb4]
25: (BlueStore::_open_db_and_around(bool)+0xc8) [0xaaaacb4dc6d8]
26: (BlueStore::_mount(bool, bool)+0x7f8) [0xaaaacb52a7d0]
27: (OSD::init()+0x20c) [0xaaaacb092bc4]
28: (main()+0x3e5c) [0xaaaacafd19d4]
29: (__libc_start_main()+0xe4) [0xffffb6370be4]
30: (()+0x5391b8) [0xaaaacb0191b8]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Files

bluefs_assert.txt (401 KB) bluefs_assert.txt

chunsong feng, 09/02/2020 12:43 AM

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by chunsong feng over 3 years ago

2020-10-17T18:02:17.658+0800 ffff7eaac980 1 bluefs _allocate failed to allocate 0x4d0000 on bdev 1, free 0x19c2108000; fallback to bdev 2
2020-10-17T18:02:17.668+0800 ffff7eaac980 1 bluefs _allocate unable to allocate 0x4d0000 on bdev 2, free 0xffffffffffffffff
2020-10-17T18:02:17.678+0800 ffff7eaac980 -1 /home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)' thread ffff7eaac980 time 2020-10-17T18:02:17.676262+0800
/home/rpmbuild/BUILD/ceph-16.0.0-4421-gcfeea548ad/src/os/bluestore/BlueFS.cc: 2198: FAILED ceph_assert(r == 0)

ceph version 16.0.0-4421-gcfeea548ad (971ac7aa5739e2dbc0df1b7c8b10b17653df1037) pacific (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0xaaaae34238e8]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xaaaae3423ad0]
 3: (BlueFS::_compact_log_async(std::unique_lock&lt;std::mutex&gt;&)+0x16d0) [0xaaaae3a42518]
 4: (BlueRocksWritableFile::Sync()+0x80) [0xaaaae3a538f0]
 5: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaaae3e8e734]
 6: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x3a0) [0xaaaae3f7c498]
 7: (rocksdb::WritableFileWriter::Sync(bool)+0x90) [0xaaaae3f7d8b8]
 8: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x274) [0xaaaae3e9f444]
 9: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1ccc) [0xaaaae3ea611c]
 10: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x30) [0xaaaae3ea63c0]
 11: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr&lt;KeyValueDB::TransactionImpl&gt;)+0x94) [0xaaaae3e5081c]
 12: (RocksDBStore::submit_transaction_sync(std::shared_ptr&lt;KeyValueDB::TransactionImpl&gt;)+0x98) [0xaaaae3e510c8]
 13: (BlueStore::_kv_sync_thread()+0x22dc) [0xaaaae39a1b04]
 14: (BlueStore::KVSyncThread::entry()+0x10) [0xaaaae39c69e0]
 15: /lib64/libpthread.so.0(+0x77f8) [0xffffa32e77f8]
 16: /lib64/libc.so.6(+0xd0edc) [0xffffa2e40edc]