Project

General

Profile

Actions
Copy link

Bug #63353

open

resharding RocksDB after upgrade to Pacific breaks OSDs

Added by Denis Polom 6 months ago. Updated 6 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Adam Kupczyk
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.2.14
ceph-qa-suite:
Pull request ID:
54331
Crash signature (v1):
Crash signature (v2):

Description

Hi

we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding) to reshard RocksDB on our OSDs.

Despite resharding reports operation as successful, OSD fails to start.

# ceph-bluestore-tool  --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard
reshard success

Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode)
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  6: (BlueStore::_mount()+0x9c) [0x5640482a896c]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  7: (OSD::init()+0x38a) [0x564047daacea]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  8: main()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  9: __libc_start_main()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  10: _start()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:      0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) **
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  in thread 7f4971ed2100 thread_name:ceph-osd
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  2: gsignal()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  3: abort()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  9: (BlueStore::_mount()+0x9c) [0x5640482a896c]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  10: (OSD::init()+0x38a) [0x564047daacea]
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  11: main()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  12: __libc_start_main()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  13: _start()
Oct 30 12:44:17 octopus2 ceph-osd[4521]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 30 12:44:17 octopus2 ceph-osd[4521]:     -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000

Any advice will be appreciated.

thx

Related issues 3 (2 open — 1 closed)

Copied to bluestore - Backport #63440: quincy: resharding RocksDB after upgrade to Pacific breaks OSDsIn ProgressKonstantin ShalyginActions
Copied to bluestore - Backport #63441: pacific: resharding RocksDB after upgrade to Pacific breaks OSDsResolvedKonstantin ShalyginActions
Copied to bluestore - Backport #63442: reef: resharding RocksDB after upgrade to Pacific breaks OSDsIn ProgressKonstantin ShalyginActions
Actions
Copy link
 #1

Updated by Adam Kupczyk 6 months ago

It is a bug. Same prefix can be inserted twice and it confuses the tool.
It should be fixed.

Please try to reshard again, but do not supply "p" twice:
  1. ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l" reshard
    If it completes, it should be fine.
    It might not complete due to some logic clash in resharding.
TIP:
There is no "l" prefix.
There is "L" prefix that is responsible for deferred writes.
I think you wanted to do:
  1. ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} L" reshard
Actions
Copy link
 #2

Updated by Joshua Baergen 6 months ago

Looks like this was a documentation bug introduced by 3879e4ad417dddef1247521c199925a679d490ac / https://github.com/ceph/ceph/pull/51771. Previously, the sharding specification was: "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P"

Actions
Copy link
 #3

Updated by Adam Kupczyk 6 months ago

  • Pull request ID set to 54331
Actions
Copy link
 #4

Updated by Laura Flores 6 months ago

  • Status changed from New to Fix Under Review
  • Backport set to quincy,pacific
Actions
Copy link
 #5

Updated by Laura Flores 6 months ago

  • Backport changed from quincy,pacific to reef,quincy,pacific
Actions
Copy link
 #6

Updated by Denis Polom 6 months ago

I can confirm - fix works

ceph-bluestore-tool \
 --path <data path> \
 --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} L P" \
 reshard
Actions
Copy link
 #7

Updated by Joshua Baergen 6 months ago

Denis: Note that 'o' should actually be 'O' (the PR missed that part of the doc fix)

Actions
Copy link
 #8

Updated by Denis Polom 6 months ago

Thanks Joshua,

that works

ceph-bluestore-tool \
 --path <data path> \
 --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \
 reshard
Actions
Copy link
 #9

Updated by Igor Fedotov 6 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #10

Updated by Backport Bot 6 months ago

  • Copied to Backport #63440: quincy: resharding RocksDB after upgrade to Pacific breaks OSDs added
Actions
Copy link
 #11

Updated by Backport Bot 6 months ago

  • Copied to Backport #63441: pacific: resharding RocksDB after upgrade to Pacific breaks OSDs added
Actions
Copy link
 #12

Updated by Backport Bot 6 months ago

  • Copied to Backport #63442: reef: resharding RocksDB after upgrade to Pacific breaks OSDs added
Actions
Copy link
 #13

Updated by Backport Bot 6 months ago

  • Tags set to backport_processed
Actions
Copy link
 #14

Updated by Konstantin Shalygin 6 months ago

  • Assignee set to Adam Kupczyk
  • Target version set to v19.0.0
Actions
Copy link

Also available in: Atom PDF