Project

General

Profile

Actions
Copy link

Bug #47874

open

Allocation error even though the block has 50 GB free

Added by Fabian Faßbender over 3 years ago. Updated over 3 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v14.2.9
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We started seeing our ceph cluster going down when OSD hit 20GB usage. Currently all out OSD have 70 GB disks attached. When trying to repair we get the errors listed below. Labels show that it has been labeled for 70 GB size. Why does bluestore not allocate over 20 GB in this case?

ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-1 --deep 1

2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken.
2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000
2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1
2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc")
 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 15: (()+0x80f12f) [0x7f8b1a20412f]
 16: (()+0x7e65) [0x7f8b18116e65]
 17: (clone()+0x6d) [0x7f8b169b588d]
2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc")

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 15: (()+0x80f12f) [0x7f8b1a20412f]
 16: (()+0x7e65) [0x7f8b18116e65]
 17: (clone()+0x6d) [0x7f8b169b588d]

*** Caught signal (Aborted) **
 in thread 7f8b115c7700 thread_name:rocksdb:low0
 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0xf5f0) [0x7f8b1811e5f0]
 2: (gsignal()+0x37) [0x7f8b168ed337]
 3: (abort()+0x148) [0x7f8b168eea28]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc]
 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 18: (()+0x80f12f) [0x7f8b1a20412f]
 19: (()+0x7e65) [0x7f8b18116e65]
 20: (clone()+0x6d) [0x7f8b169b588d]
2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) **
 in thread 7f8b115c7700 thread_name:rocksdb:low0

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0xf5f0) [0x7f8b1811e5f0]
 2: (gsignal()+0x37) [0x7f8b168ed337]
 3: (abort()+0x148) [0x7f8b168eea28]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc]
 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 18: (()+0x80f12f) [0x7f8b1a20412f]
 19: (()+0x7e65) [0x7f8b18116e65]
 20: (clone()+0x6d) [0x7f8b169b588d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  -554> 2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken.
  -553> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000
  -552> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1
  -551> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1
  -550> 2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc")

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 15: (()+0x80f12f) [0x7f8b1a20412f]
 16: (()+0x7e65) [0x7f8b18116e65]
 17: (clone()+0x6d) [0x7f8b169b588d]

  -549> 2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) **
 in thread 7f8b115c7700 thread_name:rocksdb:low0

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0xf5f0) [0x7f8b1811e5f0]
 2: (gsignal()+0x37) [0x7f8b168ed337]
 3: (abort()+0x148) [0x7f8b168eea28]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc]
 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 18: (()+0x80f12f) [0x7f8b1a20412f]
 19: (()+0x7e65) [0x7f8b18116e65]
 20: (clone()+0x6d) [0x7f8b169b588d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

   -85> 2020-10-15 18:36:11.616 7f8b23a92c00 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken.
    -4> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000
    -3> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _allocate failed to expand slow device to fit +0x40893d1
    -2> 2020-10-15 18:36:36.526 7f8b115c7700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40893d1
    -1> 2020-10-15 18:36:36.528 7f8b115c7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f8b115c7700 time 2020-10-15 18:36:36.527494
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc")

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f8b19c48ef4]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 4: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 9: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 11: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 14: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 15: (()+0x80f12f) [0x7f8b1a20412f]
 16: (()+0x7e65) [0x7f8b18116e65]
 17: (clone()+0x6d) [0x7f8b169b588d]

     0> 2020-10-15 18:36:36.530 7f8b115c7700 -1 *** Caught signal (Aborted) **
 in thread 7f8b115c7700 thread_name:rocksdb:low0

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0xf5f0) [0x7f8b1811e5f0]
 2: (gsignal()+0x37) [0x7f8b168ed337]
 3: (abort()+0x148) [0x7f8b168eea28]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f8b19c48fbc]
 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x5607c086bb77]
 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x5607c086bd6b]
 7: (BlueRocksWritableFile::Flush()+0x3d) [0x5607c0a28b7d]
 8: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x5607c0c5005c]
 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x5607c0c5021e]
 10: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0x89d) [0x5607c0c948ad]
 11: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x597) [0x5607c0c96397]
 12: (rocksdb::CompactionJob::Run()+0x274) [0x5607c0c97564]
 13: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd19) [0x5607c0afdaa9]
 14: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd5) [0x5607c0b03f75]
 15: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x3a) [0x5607c0b0446a]
 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a) [0x5607c0cd68aa]
 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x5d) [0x5607c0cd6a3d]
 18: (()+0x80f12f) [0x7f8b1a20412f]
 19: (()+0x7e65) [0x7f8b18116e65]
 20: (clone()+0x6d) [0x7f8b169b588d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ceph-volume lvm list

====== osd.1 =======

  [block]       /dev/ceph-27c26f83-acb7-45dc-a43e-7650fde0eb86/osd-data-5413cbbf-b31c-4990-b4da-4a971343fbae

      block device              /dev/ceph-27c26f83-acb7-45dc-a43e-7650fde0eb86/osd-data-5413cbbf-b31c-4990-b4da-4a971343fbae
      block uuid                q7bhgE-J2w3-Kf3V-Hk3a-gA7c-3sJK-BAkkZi
      cephx lockbox secret      
      cluster fsid              1bfe0501-419e-4746-bde5-00b27b1c6f70
      cluster name              ceph
      crush device class        None
      encrypted                 0
      osd fsid                  b44c79ea-5c91-4cfb-aa3b-9bd5f8ae3870
      osd id                    1
      type                      block
      vdo                       0
      devices                   /dev/sda2

ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-1

... 90000] = 0x4d8ec0000 : using 0x4d5110000(19 GiB)

ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 --deep 1

inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-1/block": {
        "osd_uuid": "b44c79ea-5c91-4cfb-aa3b-9bd5f8ae3870",
        "size": 75157733376,
        "btime": "2020-05-04 18:27:51.128807",
        "description": "main",
        "bluefs": "1",
        "ceph_fsid": "1bfe0501-419e-4746-bde5-00b27b1c6f70",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQCaXrBeG/uCCxAACvb5xFczsCNPqJ014O5qMA==",
        "ready": "ready",
        "require_osd_release": "14",
        "whoami": "1" 
    }
}

Actions
Copy link
 #1

Updated by Igor Fedotov over 3 years ago

Hi Fabian,
Bluestore is unable to allocate space for additional BlueFS data. This line:
2020-10-15 18:36:36.526 7f8b115c7700 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x4090000 min_size 0x4090000 > allocated total 0x2dc0000 bluefs_shared_alloc_size 0x10000 allocated 0x2dc0000 available 0x 800b000

indicates just 0x800b000 of free space at main device.

Afair bluefs-bdev-sizes command reports BLUEFS part of disk usage only. Hence for main device it doesn't take user data into account. Perhaps the rest of space is allocated for that data. Isn't this a clue?

You might want to inspect bluestore_allocated performance counter to learn actual allocated space at main device.

Actions
Copy link
 #2

Updated by Neha Ojha over 3 years ago

  • Status changed from New to Need More Info
Actions
Copy link

Also available in: Atom PDF