Bug #63353
openresharding RocksDB after upgrade to Pacific breaks OSDs
0%
Description
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
Any advice will be appreciated.
thx
Updated by Adam Kupczyk 6 months ago
It is a bug. Same prefix can be inserted twice and it confuses the tool.
It should be fixed.
- ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l" reshard
If it completes, it should be fine.
It might not complete due to some logic clash in resharding.
There is no "l" prefix.
There is "L" prefix that is responsible for deferred writes.
I think you wanted to do:
- ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} L" reshard
Updated by Joshua Baergen 6 months ago
Looks like this was a documentation bug introduced by 3879e4ad417dddef1247521c199925a679d490ac / https://github.com/ceph/ceph/pull/51771. Previously, the sharding specification was: "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P"
Updated by Laura Flores 6 months ago
- Status changed from New to Fix Under Review
- Backport set to quincy,pacific
Updated by Laura Flores 6 months ago
- Backport changed from quincy,pacific to reef,quincy,pacific
Updated by Denis Polom 6 months ago
I can confirm - fix works
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Updated by Joshua Baergen 6 months ago
Denis: Note that 'o' should actually be 'O' (the PR missed that part of the doc fix)
Updated by Denis Polom 6 months ago
Thanks Joshua,
that works
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Updated by Igor Fedotov 6 months ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 6 months ago
- Copied to Backport #63440: quincy: resharding RocksDB after upgrade to Pacific breaks OSDs added
Updated by Backport Bot 6 months ago
- Copied to Backport #63441: pacific: resharding RocksDB after upgrade to Pacific breaks OSDs added
Updated by Backport Bot 6 months ago
- Copied to Backport #63442: reef: resharding RocksDB after upgrade to Pacific breaks OSDs added
Updated by Konstantin Shalygin 6 months ago
- Assignee set to Adam Kupczyk
- Target version set to v19.0.0