Bug #51445
cannot enable osd resharding on pacific
% Done:
0%
Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
when following the docs at https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#bluestore-rocksdb-sharding
for enabling sharding (incl pacific defaults) on an upgraded cluster by executing
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
the tools prints some stack traces and the osd is broken:
2021-06-29T07:39:40.949-0500 7f54703dd240 -1 rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} ceph-bluestore-tool: /build/ceph/ceph-16.2.4/src/rocksdb/db/column_family.cc:1387: rocksdb::ColumnFamilySet::~ColumnFamilySet(): Assertion `last_ref' failed. *** Caught signal (Aborted) ** in thread 7f54703dd240 thread_name:ceph-bluestore- ceph version 16.2.4 (a912ff2c95b1f9a8e2e48509e602ee008d5c9434) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f5470aa5140] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x2540f) [0x7f54705be40f] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x34662) [0x7f54705cd662] 6: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x82) [0x55ec0217fb36] 7: (std::default_delete<rocksdb::ColumnFamilySet>::operator()(rocksdb::ColumnFamilySet*) const+0x22) [0x55ec01fd699c] 8: (std::__uniq_ptr_impl<rocksdb::ColumnFamilySet, std::default_delete<rocksdb::ColumnFamilySet> >::reset(rocksdb::ColumnFamilySet*)+0x5b) [0x55ec01fd6de5] 9: (std::unique_ptr<rocksdb::ColumnFamilySet, std::default_delete<rocksdb::ColumnFamilySet> >::reset(rocksdb::ColumnFamilySet*)+0x2f) [0x55ec01fd08f5] 10: (rocksdb::VersionSet::~VersionSet()+0x4f) [0x55ec01fb6ff9] 11: (rocksdb::VersionSet::~VersionSet()+0x18) [0x55ec01fb7170] 12: (std::default_delete<rocksdb::VersionSet>::operator()(rocksdb::VersionSet*) const+0x28) [0x55ec01e68d64] 13: (std::__uniq_ptr_impl<rocksdb::VersionSet, std::default_delete<rocksdb::VersionSet> >::reset(rocksdb::VersionSet*)+0x5b) [0x55ec01e6ac81] 14: (std::unique_ptr<rocksdb::VersionSet, std::default_delete<rocksdb::VersionSet> >::reset(rocksdb::VersionSet*)+0x2f) [0x55ec01e5bef5] 15: (rocksdb::DBImpl::CloseHelper()+0xa12) [0x55ec01e27414] 16: (rocksdb::DBImpl::~DBImpl()+0x4e) [0x55ec01e2784a] 17: (rocksdb::DBImpl::~DBImpl()+0x18) [0x55ec01e27bfa] 18: (RocksDBStore::close()+0x355) [0x55ec01dfc9a5] 19: (RocksDBStore::reshard(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RocksDBStore::resharding_ctrl const*)+0x231) [0x55ec01e03ec1] 20: main() 21: __libc_start_main() 22: _start() 2021-06-29T07:39:40.965-0500 7f54703dd240 -1 *** Caught signal (Aborted) ** in thread 7f54703dd240 thread_name:ceph-bluestore-
it seems that the rocksdb option parser does not understand the 'block_cache={type=binned_lru}' option despite it being the default for new osds on pacific
user reported it first here:
https://forum.proxmox.com/threads/cant-reshard-osds-under-ceph-pacific-16-2-4.91630/
Related issues
History
#1 Updated by Josh Durgin over 2 years ago
- Project changed from Ceph to bluestore
#2 Updated by Neha Ojha over 2 years ago
- Assignee set to Adam Kupczyk
#3 Updated by Neha Ojha over 2 years ago
- Duplicated by Bug #51676: rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} error resharding: (22) Invalid argument added
#4 Updated by Neha Ojha over 2 years ago
- Priority changed from Normal to Urgent
#5 Updated by Neha Ojha over 2 years ago
- Status changed from New to Fix Under Review
- Backport set to pacific
- Pull request ID set to 42345
#6 Updated by Yuri Weinstein over 2 years ago
#7 Updated by Neha Ojha over 2 years ago
- Status changed from Fix Under Review to Pending Backport
#8 Updated by Backport Bot over 2 years ago
- Copied to Backport #52246: pacific: cannot enable osd resharding on pacific added
#9 Updated by Igor Fedotov over 2 years ago
- Status changed from Pending Backport to Resolved