Actions
Bug #47874
openAllocation error even though the block has 50 GB free
Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We started seeing our ceph cluster going down when OSD hit 20GB usage. Currently all out OSD have 70 GB disks attached. When trying to repair we get the errors listed below. Labels show that it has been labeled for 70 GB size. Why does bluestore not allocate over 20 GB in this case?
ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-1 --deep 1
2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 15: (()+0x80f12f) [0x7f8b1a20412f] 16: (()+0x7e65) [0x7f8b18116e65] 17: (clone()+0x6d) [0x7f8b169b588d] 2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 15: (()+0x80f12f) [0x7f8b1a20412f] 16: (()+0x7e65) [0x7f8b18116e65] 17: (clone()+0x6d) [0x7f8b169b588d] *** Caught signal (Aborted) ** in thread 7f8b115c7700 thread_name:rocksdb:low0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0xf5f0) [0x7f8b1811e5f0] 2: (gsignal()+0x37) [0x7f8b168ed337] 3: (abort()+0x148) [0x7f8b168eea28] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 18: (()+0x80f12f) [0x7f8b1a20412f] 19: (()+0x7e65) [0x7f8b18116e65] 20: (clone()+0x6d) [0x7f8b169b588d] 2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) ** in thread 7f8b115c7700 thread_name:rocksdb:low0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0xf5f0) [0x7f8b1811e5f0] 2: (gsignal()+0x37) [0x7f8b168ed337] 3: (abort()+0x148) [0x7f8b168eea28] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 18: (()+0x80f12f) [0x7f8b1a20412f] 19: (()+0x7e65) [0x7f8b18116e65] 20: (clone()+0x6d) [0x7f8b169b588d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -554> 2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. -553> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000 -552> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1 -551> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1 -550> 2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 15: (()+0x80f12f) [0x7f8b1a20412f] 16: (()+0x7e65) [0x7f8b18116e65] 17: (clone()+0x6d) [0x7f8b169b588d] -549> 2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) ** in thread 7f8b115c7700 thread_name:rocksdb:low0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0xf5f0) [0x7f8b1811e5f0] 2: (gsignal()+0x37) [0x7f8b168ed337] 3: (abort()+0x148) [0x7f8b168eea28] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 18: (()+0x80f12f) [0x7f8b1a20412f] 19: (()+0x7e65) [0x7f8b18116e65] 20: (clone()+0x6d) [0x7f8b169b588d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -85> 2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. -4> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000 -3> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1 -2> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1 -1> 2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 15: (()+0x80f12f) [0x7f8b1a20412f] 16: (()+0x7e65) [0x7f8b18116e65] 17: (clone()+0x6d) [0x7f8b169b588d] 0> 2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) ** in thread 7f8b115c7700 thread_name:rocksdb:low0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0xf5f0) [0x7f8b1811e5f0] 2: (gsignal()+0x37) [0x7f8b168ed337] 3: (abort()+0x148) [0x7f8b168eea28] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b] 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d] 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c] 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e] 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad] 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397] 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564] 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9] 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75] 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a] 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa] 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d] 18: (()+0x80f12f) [0x7f8b1a20412f] 19: (()+0x7e65) [0x7f8b18116e65] 20: (clone()+0x6d) [0x7f8b169b588d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-volume lvm list
====== osd.1 ======= [block] /dev/ceph-27c26f83-acb7-45dc-a43e-7650fde0eb86/osd-data-5413cbbf-b31c-4990-b4da-4a971343fbae block device /dev/ceph-27c26f83-acb7-45dc-a43e-7650fde0eb86/osd-data-5413cbbf-b31c-4990-b4da-4a971343fbae block uuid q7bhgE-J2w3-Kf3V-Hk3a-gA7c-3sJK-BAkkZi cephx lockbox secret cluster fsid 1bfe0501-419e-4746-bde5-00b27b1c6f70 cluster name ceph crush device class None encrypted 0 osd fsid b44c79ea-5c91-4cfb-aa3b-9bd5f8ae3870 osd id 1 type block vdo 0 devices /dev/sda2
ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-1
... 90000] = 0x4d8ec0000 : using 0x4d5110000(19 GiB)
ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 --deep 1
inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "b44c79ea-5c91-4cfb-aa3b-9bd5f8ae3870", "size": 75157733376, "btime": "2020-05-04 18:27:51.128807", "description": "main", "bluefs": "1", "ceph_fsid": "1bfe0501-419e-4746-bde5-00b27b1c6f70", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCaXrBeG/uCCxAACvb5xFczsCNPqJ014O5qMA==", "ready": "ready", "require_osd_release": "14", "whoami": "1" } }
Actions