Bug #53590: ceph abort at bluefs enospc - bluestore - Ceph

Actions

Copy link

Bug #53590

closed

ceph abort at bluefs enospc

Added by chunsong feng over 2 years ago. Updated over 2 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

Ceph - v16.2.7

% Done:

Source:

Community (dev)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

-14> 2021-12-12T14:31:17.268+0800 fffc90155480 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1639290677272593, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30297, "file_size": 6602583, "table_properties": {"data_size": 6068034, "index_size": 42547, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 491141, "raw_key_size": 3535776, "raw_average_key_size": 18, "raw_value_size": 3142912, "raw_average_value_size": 16, "num_data_blocks": 1489, "num_entries": 196432, "num_deletions": 0, "num_merge_operands": 196432, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": ".T:int64_array.b:bitwise_xor", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1639290675, "oldest_key_time": 3, "file_creation_time": 0}}
-13> 2021-12-12T14:31:17.416+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30283 mode 2
-12> 2021-12-12T14:31:17.520+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30284 mode 2
-11> 2021-12-12T14:31:17.572+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30285 mode 2
-10> 2021-12-12T14:31:17.728+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30286 mode 2
-9> 2021-12-12T14:31:18.004+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30288 mode 2
-8> 2021-12-12T14:31:18.412+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30290 mode 2
-7> 2021-12-12T14:31:19.436+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30293 mode 2
-6> 2021-12-12T14:31:20.431+0800 fffc90155480 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1639290680434378, "cf_name": "P", "job": 1, "event": "table_file_creation", "file_number": 30298, "file_size": 2601202, "table_properties": {"data_size": 2345733, "index_size": 32807, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 221829, "raw_key_size": 4239371, "raw_average_key_size": 47, "raw_value_size": 1008510, "raw_average_value_size": 11, "num_data_blocks": 581, "num_entries": 88632, "num_deletions": 84777, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "P", "column_family_id": 11, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1639290679, "oldest_key_time": 3, "file_creation_time": 0}}
-5> 2021-12-12T14:31:20.443+0800 fffc90155480 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #30295 mode 2
-4> 2021-12-12T14:31:23.655+0800 fffc90155480 1 bluefs _allocate unable to allocate 0x100000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x3e7fc00000, block size 0x1000, free 0x137bfaa000, fragmentation 0.580174, allocated 0x0
-3> 2021-12-12T14:31:23.655+0800 fffc90155480 -1 bluefs _allocate allocation failed, needed 0xf9717
-2> 2021-12-12T14:31:23.655+0800 fffc90155480 -1 bluefs _flush_range allocated: 0xc00000 offset: 0xbf9fef length: 0xff728
-1> 2021-12-12T14:31:23.675+0800 fffc90155480 -1 /home/rpmbuild/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread fffc90155480 time 2021-12-12T14:31:23.660647+0800
/home/rpmbuild/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc")

ceph version 16.2.6-236-gf3f9543f9dc (f3f9543f9dcfc57e5aa85a161c84bc49ee594ec1) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xec) [0xaaae53a9bf04]
2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1100) [0xaaae540da730]
3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0xac) [0xaaae540daa94]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x38) [0xaaae540ea9f0]
5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x110) [0xaaae54102668]
6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaae5455fecc]
7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x3b8) [0xaaae5464df78]
8: (rocksdb::WritableFileWriter::Flush()+0x1d4) [0xaaae5464ea4c]
9: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaae5464f340]
10: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaae5474cb20]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaae5474d26c]
12: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaae5474d808]
13: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaae5474d884]
14: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaae54750564]
15: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaae547056b8]
16: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaae545983ac]
17: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaae54599ca8]
18: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaae5459b414]
19: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaae545962cc]
20: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaae545973a4]
21: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xca0) [0xaaae5451e7a0]
22: (BlueStore::_open_db(bool, bool, bool)+0x80c) [0xaaae53fdf904]
23: (BlueStore::_open_db_and_around(bool, bool)+0x2cc) [0xaaae540447b4]
24: (BlueStore::_mount()+0x218) [0xaaae54047668]
25: (OSD::init()+0x20c) [0xaaae53b68b1c]
26: main()
27: __libc_start_main()
28: /usr/bin/ceph-osd(+0x64c674) [0xaaae53aec674]

0> 2021-12-12T14:31:23.695+0800 fffc90155480 -1 ** Caught signal (Aborted) *
 in thread fffc90155480 thread_name:ceph-osd

ceph version 16.2.6-236-gf3f9543f9dc (f3f9543f9dcfc57e5aa85a161c84bc49ee594ec1) pacific (stable)
1: _kernel_rt_sigreturn()
2: gsignal()
3: abort()
4: (ceph::_ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1a8) [0xaaae53a9bfc0]
5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1100) [0xaaae540da730]
6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0xac) [0xaaae540daa94]
7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x38) [0xaaae540ea9f0]
8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x110) [0xaaae54102668]
9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x2c) [0xaaae5455fecc]
10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x3b8) [0xaaae5464df78]
11: (rocksdb::WritableFileWriter::Flush()+0x1d4) [0xaaae5464ea4c]
12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2a8) [0xaaae5464f340]
13: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xa0) [0xaaae5474cb20]
14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaae5474d26c]
15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaae5474d808]
16: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaae5474d884]
17: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2bc) [0xaaae54750564]
18: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x768) [0xaaae547056b8]
19: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaae545983ac]
20: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0xf40) [0xaaae54599ca8]
21: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaae5459b414]
22: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*, bool, bool)+0x464) [0xaaae545962cc]
23: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >, rocksdb::DB*)+0x1c) [0xaaae545973a4]
24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xca0) [0xaaae5451e7a0]
25: (BlueStore::_open_db(bool, bool, bool)+0x80c) [0xaaae53fdf904]
26: (BlueStore::_open_db_and_around(bool, bool)+0x2cc) [0xaaae540447b4]
27: (BlueStore::_mount()+0x218) [0xaaae54047668]
28: (OSD::init()+0x20c) [0xaaae53b68b1c]
29: main()
30: __libc_start_main()
31: /usr/bin/ceph-osd(+0x64c674) [0xaaae53aec674]