Mar 28 23:51:43 backupfs-2 systemd[1]: ceph-osd-12.service: Main process exited, code=dumped, status=6/ABRT
Mar 28 23:51:43 backupfs-2 systemd[1]: ceph-osd-12.service: Failed with result 'core-dump'.
Mar 28 23:51:43 backupfs-2 systemd[1]: ceph-osd-12.service: Consumed 2min 35.273s CPU time, received 6.3K IP traffic, sent 4.4K IP traffic.
Mar 28 23:51:43 backupfs-2 systemd[1]: ceph-osd-12.service: Scheduled restart job, restart counter is at 1.
Mar 28 23:51:43 backupfs-2 systemd[1]: Stopped Ceph OSD.
Mar 28 23:51:43 backupfs-2 systemd[1]: ceph-osd-12.service: Consumed 2min 35.273s CPU time, received 6.3K IP traffic, sent 4.4K IP traffic.
Mar 28 23:51:43 backupfs-2 systemd[1]: Starting Ceph OSD...
Mar 28 23:51:43 backupfs-2 systemd[1]: Started Ceph OSD.
Mar 28 23:51:44 backupfs-2 ceph-osd[864328]: 2024-03-28T23:51:44.108+0000 7fcf3ca7ee80 -1 Falling back to public interface
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _allocate allocation failed, needed 0x80000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _flush_range_F allocated: 0x383000 offset: 0x3824da length: 0x805b1
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range_F(FileWriter*, uint64_t, uint64_t)' thread 7fcf3ca7ee80 time 2024-03-28T23:55:26.072920+0000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: 3396: ceph_abort_msg("bluefs enospc")
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdc) [0x5642b4dbcdba]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: *** Caught signal (Aborted) **
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  in thread 7fcf3ca7ee80 thread_name:ceph-osd
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: 2024-03-28T23:55:26.077+0000 7fcf3ca7ee80 -1 /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range_F(FileWriter*, uint64_t, uint64_t)' thread 7fcf3ca7ee80 time 2024-03-28T23:55:26.072920+0000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: 3396: ceph_abort_msg("bluefs enospc")
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdc) [0x5642b4dbcdba]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x3deb0) [0x7fcf3c654eb0]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x8d02c) [0x7fcf3c6a402c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: raise()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: abort()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x19d) [0x5642b4dbce7b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  28: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  29: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  30: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  31: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: 2024-03-28T23:55:26.081+0000 7fcf3ca7ee80 -1 *** Caught signal (Aborted) **
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  in thread 7fcf3ca7ee80 thread_name:ceph-osd
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x3deb0) [0x7fcf3c654eb0]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x8d02c) [0x7fcf3c6a402c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: raise()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: abort()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x19d) [0x5642b4dbce7b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  28: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  29: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  30: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  31: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2535> 2024-03-28T23:51:44.108+0000 7fcf3ca7ee80 -1 Falling back to public interface
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:     -3> 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _allocate allocation failed, needed 0x80000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:     -2> 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _flush_range_F allocated: 0x383000 offset: 0x3824da length: 0x805b1
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:     -1> 2024-03-28T23:55:26.077+0000 7fcf3ca7ee80 -1 /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range_F(FileWriter*, uint64_t, uint64_t)' thread 7fcf3ca7ee80 time 2024-03-28T23:55:26.072920+0000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: 3396: ceph_abort_msg("bluefs enospc")
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdc) [0x5642b4dbcdba]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:      0> 2024-03-28T23:55:26.081+0000 7fcf3ca7ee80 -1 *** Caught signal (Aborted) **
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  in thread 7fcf3ca7ee80 thread_name:ceph-osd
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x3deb0) [0x7fcf3c654eb0]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x8d02c) [0x7fcf3c6a402c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: raise()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: abort()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x19d) [0x5642b4dbce7b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  28: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  29: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  30: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  31: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2764> 2024-03-28T23:51:44.108+0000 7fcf3ca7ee80 -1 Falling back to public interface
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2763> 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _allocate allocation failed, needed 0x80000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2762> 2024-03-28T23:55:26.072+0000 7fcf3ca7ee80 -1 bluefs _flush_range_F allocated: 0x383000 offset: 0x3824da length: 0x805b1
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2761> 2024-03-28T23:55:26.077+0000 7fcf3ca7ee80 -1 /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range_F(FileWriter*, uint64_t, uint64_t)' thread 7fcf3ca7ee80 time 2024-03-28T23:55:26.072920+0000
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]: /build/ceph-18.2.1/src/os/bluestore/BlueFS.cc: 3396: ceph_abort_msg("bluefs enospc")
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdc) [0x5642b4dbcdba]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  -2760> 2024-03-28T23:55:26.081+0000 7fcf3ca7ee80 -1 *** Caught signal (Aborted) **
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  in thread 7fcf3ca7ee80 thread_name:ceph-osd
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable)
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  1: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x3deb0) [0x7fcf3c654eb0]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  2: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x8d02c) [0x7fcf3c6a402c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  3: raise()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  4: abort()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x19d) [0x5642b4dbce7b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  6: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x908) [0x5642b54ff2f8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  7: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5642b54ff400]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  8: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0xd6) [0x5642b55116e6]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  9: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x1b) [0x5642b552de4b]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  10: /nix/store/fhqil7cb11jb5ckrkayjllsm173agiyd-rocksdb-7.9.2/lib/librocksdb.so.7(+0x590a56) [0x7fcf3d790a56]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  11: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x3fc) [0x7fcf3d7e696c]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0x6fe) [0x7fcf3d7ec9be]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  13: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1f5) [0x7fcf3d86a7c5]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x124) [0x7fcf3d86af14]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x9f) [0x7fcf3d86c34f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2c7) [0x7fcf3d86e8e7]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  17: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned >
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x112b) [0x7fcf3d6308bb]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  19: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0x1d15) [0x7fcf3d633825]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  20: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1891) [0x7fcf3d635ab1]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x7df) [0x7fcf3d62c43f]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x7fcf3d62e6f4]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb43) [0x5642b5a96253]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  24: (BlueStore::_open_db(bool, bool, bool)+0x658) [0x5642b544a4e8]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  25: (BlueStore::_open_db_and_around(bool, bool)+0x481) [0x5642b5493711]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  26: (BlueStore::_mount()+0x38d) [0x5642b549648d]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  27: (OSD::init()+0x4f1) [0x5642b4edd881]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  28: main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  29: /nix/store/7jiqcrg061xi5clniy7z5pvkc4jiaqav-glibc-2.38-27/lib/libc.so.6(+0x280ce) [0x7fcf3c63f0ce]
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  30: __libc_start_main()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  31: _start()
Mar 28 23:55:26 backupfs-2 ceph-osd[864328]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 28 23:55:29 backupfs-2 systemd[1]: ceph-osd-12.service: Main process exited, code=dumped, status=6/ABRT
Mar 28 23:55:29 backupfs-2 systemd[1]: ceph-osd-12.service: Failed with result 'core-dump'.