Project

General

Profile

Bug #51445

cannot enable osd resharding on pacific

Added by Dominik Csapak over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

when following the docs at https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#bluestore-rocksdb-sharding
for enabling sharding (incl pacific defaults) on an upgraded cluster by executing

ceph-bluestore-tool \
  --path <data path> \
  --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \
  reshard

the tools prints some stack traces and the osd is broken:

2021-06-29T07:39:40.949-0500 7f54703dd240 -1 rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru}
ceph-bluestore-tool: /build/ceph/ceph-16.2.4/src/rocksdb/db/column_family.cc:1387: rocksdb::ColumnFamilySet::~ColumnFamilySet(): Assertion `last_ref' failed.
*** Caught signal (Aborted) **
 in thread 7f54703dd240 thread_name:ceph-bluestore-
 ceph version 16.2.4 (a912ff2c95b1f9a8e2e48509e602ee008d5c9434) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f5470aa5140]
 2: gsignal()
 3: abort()
 4: /lib/x86_64-linux-gnu/libc.so.6(+0x2540f) [0x7f54705be40f]
 5: /lib/x86_64-linux-gnu/libc.so.6(+0x34662) [0x7f54705cd662]
 6: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x82) [0x55ec0217fb36]
 7: (std::default_delete<rocksdb::ColumnFamilySet>::operator()(rocksdb::ColumnFamilySet*) const+0x22) [0x55ec01fd699c]
 8: (std::__uniq_ptr_impl<rocksdb::ColumnFamilySet, std::default_delete<rocksdb::ColumnFamilySet> >::reset(rocksdb::ColumnFamilySet*)+0x5b) [0x55ec01fd6de5]
 9: (std::unique_ptr<rocksdb::ColumnFamilySet, std::default_delete<rocksdb::ColumnFamilySet> >::reset(rocksdb::ColumnFamilySet*)+0x2f) [0x55ec01fd08f5]
 10: (rocksdb::VersionSet::~VersionSet()+0x4f) [0x55ec01fb6ff9]
 11: (rocksdb::VersionSet::~VersionSet()+0x18) [0x55ec01fb7170]
 12: (std::default_delete<rocksdb::VersionSet>::operator()(rocksdb::VersionSet*) const+0x28) [0x55ec01e68d64]
 13: (std::__uniq_ptr_impl<rocksdb::VersionSet, std::default_delete<rocksdb::VersionSet> >::reset(rocksdb::VersionSet*)+0x5b) [0x55ec01e6ac81]
 14: (std::unique_ptr<rocksdb::VersionSet, std::default_delete<rocksdb::VersionSet> >::reset(rocksdb::VersionSet*)+0x2f) [0x55ec01e5bef5]
 15: (rocksdb::DBImpl::CloseHelper()+0xa12) [0x55ec01e27414]
 16: (rocksdb::DBImpl::~DBImpl()+0x4e) [0x55ec01e2784a]
 17: (rocksdb::DBImpl::~DBImpl()+0x18) [0x55ec01e27bfa]
 18: (RocksDBStore::close()+0x355) [0x55ec01dfc9a5]
 19: (RocksDBStore::reshard(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RocksDBStore::resharding_ctrl const*)+0x231) [0x55ec01e03ec1]
 20: main()
 21: __libc_start_main()
 22: _start()
2021-06-29T07:39:40.965-0500 7f54703dd240 -1 *** Caught signal (Aborted) **
 in thread 7f54703dd240 thread_name:ceph-bluestore-

it seems that the rocksdb option parser does not understand the 'block_cache={type=binned_lru}' option despite it being the default for new osds on pacific

user reported it first here:
https://forum.proxmox.com/threads/cant-reshard-osds-under-ceph-pacific-16-2-4.91630/


Related issues

Duplicated by bluestore - Bug #51676: rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} error resharding: (22) Invalid argument Duplicate
Copied to bluestore - Backport #52246: pacific: cannot enable osd resharding on pacific Resolved

History

#1 Updated by Josh Durgin over 2 years ago

  • Project changed from Ceph to bluestore

#2 Updated by Neha Ojha over 2 years ago

  • Assignee set to Adam Kupczyk

#3 Updated by Neha Ojha over 2 years ago

  • Duplicated by Bug #51676: rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} error resharding: (22) Invalid argument added

#4 Updated by Neha Ojha over 2 years ago

  • Priority changed from Normal to Urgent

#5 Updated by Neha Ojha over 2 years ago

  • Status changed from New to Fix Under Review
  • Backport set to pacific
  • Pull request ID set to 42345

#7 Updated by Neha Ojha over 2 years ago

  • Status changed from Fix Under Review to Pending Backport

#8 Updated by Backport Bot over 2 years ago

  • Copied to Backport #52246: pacific: cannot enable osd resharding on pacific added

#9 Updated by Igor Fedotov over 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF