Ceph » bluestore

Hi,

I have an octopus 15.2.14 deployment and I'd like to remove all my db devices from the nvme drive and keep together with the block.
The osds has been created with dmcrypt.
The command that I'm using for this device:

ceph-volume lvm migrate --osd-id 48 --osd-fsid `cat /var/lib/ceph/osd/ceph-$num/fsid` --from db --target ceph-4f6e1d90-22ca-41c8-9fe4-60b029eb74e4/osd-block-4df35929-9b26-4d7e-9eb2-ae884c7d71cf
--> Migrate to existing, Source: ['--devs-source', '/var/lib/ceph/osd/ceph-48/block.db'] Target: /var/lib/ceph/osd/ceph-48/block
--> Migration successful.

====== osd.48 ======

  [db]          /dev/ceph-3e291477-42b4-4bd5-bb73-eadd075e4f38/osd-db-c71df794-f8d8-4691-b84b-fd62cf22d2e8

      block device              /dev/ceph-4f6e1d90-22ca-41c8-9fe4-60b029eb74e4/osd-block-4df35929-9b26-4d7e-9eb2-ae884c7d71cf
      block uuid                vHOcEe-wrFp-DkHt-fUbq-1suN-xSvO-raUwgJ
      cephx lockbox secret      AQAA30Rh52NPBRAAJfHCkp5CceqxxFzt5P/6Xw==
      cluster fsid              5a07ec50-4eee-4336-aa11-46ca76edcc24
      cluster name              ceph
      crush device class        None
      db device                 /dev/ceph-3e291477-42b4-4bd5-bb73-eadd075e4f38/osd-db-c71df794-f8d8-4691-b84b-fd62cf22d2e8
      db uuid                   NKjT2D-TRUz-xbKc-zL2q-YHIN-NnUq-LvXgx2
      encrypted                 1
      osd fsid                  4df35929-9b26-4d7e-9eb2-ae884c7d71cf
      osd id                    48
      osdspec affinity
      type                      db
      vdo                       0
      devices                   /dev/nvme0n1

  [block]       /dev/ceph-4f6e1d90-22ca-41c8-9fe4-60b029eb74e4/osd-block-4df35929-9b26-4d7e-9eb2-ae884c7d71cf

      block device              /dev/ceph-4f6e1d90-22ca-41c8-9fe4-60b029eb74e4/osd-block-4df35929-9b26-4d7e-9eb2-ae884c7d71cf
      block uuid                vHOcEe-wrFp-DkHt-fUbq-1suN-xSvO-raUwgJ
      cephx lockbox secret      AQAA30Rh52NPBRAAJfHCkp5CceqxxFzt5P/6Xw==
      cluster fsid              5a07ec50-4eee-4336-aa11-46ca76edcc24
      cluster name              ceph
      crush device class        None
      db device                 /dev/ceph-3e291477-42b4-4bd5-bb73-eadd075e4f38/osd-db-c71df794-f8d8-4691-b84b-fd62cf22d2e8
      db uuid                   NKjT2D-TRUz-xbKc-zL2q-YHIN-NnUq-LvXgx2
      encrypted                 1
      osd fsid                  4df35929-9b26-4d7e-9eb2-ae884c7d71cf
      osd id                    48
      osdspec affinity
      type                      block
      vdo                       0
      devices                   /dev/sdf

And after coredumped and never started back again.

The backtrace that I get:

{
    "backtrace": [
        "(()+0x12b20) [0x7f310aa49b20]",
        "(gsignal()+0x10f) [0x7f31096aa37f]",
        "(abort()+0x127) [0x7f3109694db5]",
        "(()+0x9009b) [0x7f310a06209b]",
        "(()+0x9653c) [0x7f310a06853c]",
        "(()+0x95559) [0x7f310a067559]",
        "(__gxx_personality_v0()+0x2a8) [0x7f310a067ed8]",
        "(()+0x10b03) [0x7f3109a48b03]",
        "(_Unwind_RaiseException()+0x2b1) [0x7f3109a49071]",
        "(__cxa_throw()+0x3b) [0x7f310a0687eb]",
        "(()+0x19fa4) [0x7f310b7b6fa4]",
        "(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7f310b7d8c96]",
        "(()+0x10d0f8e) [0x55ffa520df8e]",
        "(rocksdb::Version::~Version()+0x104) [0x55ffa521d174]",
        "(rocksdb::Version::Unref()+0x21) [0x55ffa521d221]",
        "(rocksdb::ColumnFamilyData::~ColumnFamilyData()+0x5a) [0x55ffa52efcca]",
        "(rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x88) [0x55ffa52f0568]",
        "(rocksdb::VersionSet::~VersionSet()+0x5e) [0x55ffa520e01e]",
        "(rocksdb::VersionSet::~VersionSet()+0x11) [0x55ffa520e261]",
        "(rocksdb::DBImpl::CloseHelper()+0x616) [0x55ffa5155ed6]",
        "(rocksdb::DBImpl::~DBImpl()+0x83b) [0x55ffa515c35b]",
        "(rocksdb::DBImplReadOnly::~DBImplReadOnly()+0x11) [0x55ffa51a3bc1]",
        "(rocksdb::DB::OpenForReadOnly(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x1089) [0x55ffa51a57e9]",
        "(RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x14ca) [0x55ffa51285ca]",
        "(BlueStore::_open_db(bool, bool, bool)+0x1314) [0x55ffa4bc27e4]",
        "(BlueStore::_open_db_and_around(bool)+0x4c) [0x55ffa4bd4c5c]",
        "(BlueStore::_mount(bool, bool)+0x847) [0x55ffa4c2e047]",
        "(OSD::init()+0x380) [0x55ffa4753a70]",
        "(main()+0x47f1) [0x55ffa46a6901]",
        "(__libc_start_main()+0xf3) [0x7f3109696493]",
        "(_start()+0x2e) [0x55ffa46d4e3e]" 
    ],
    "ceph_version": "15.2.14",
    "crash_id": "2021-10-05T13:31:28.513463Z_b6818598-4960-4ed6-942a-d4a7ff37a758",
    "entity_name": "osd.48",
    "os_id": "centos",
    "os_name": "CentOS Linux",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-osd",
    "stack_sig": "6a43b6c219adac393b239fbea4a53ff87c4185bcd213724f0d721b452b81ddbf",
    "timestamp": "2021-10-05T13:31:28.513463Z",
    "utsname_hostname": "server-2s07",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-305.19.1.el8_4.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Wed Sep 15 15:39:39 UTC 2021" 
}

In the logs there are entries like this:

   -82> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:        Options.write_buffer_size: 268435456
   -81> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:  Options.max_write_buffer_number: 4
   -80> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:          Options.compression: NoCompression
   -79> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                  Options.bottommost_compression: Disabled
   -78> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:       Options.prefix_extractor: nullptr
   -77> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
   -76> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:             Options.num_levels: 7
   -75> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
   -74> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
   -73> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:            Options.bottommost_compression_opts.window_bits: -14
   -72> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                  Options.bottommost_compression_opts.level: 32767
   -71> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:               Options.bottommost_compression_opts.strategy: 0
   -70> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:         Options.bottommost_compression_opts.max_dict_bytes: 0
   -69> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:         Options.bottommost_compression_opts.zstd_max_train_bytes: 0
   -68> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                  Options.bottommost_compression_opts.enabled: false
   -67> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:            Options.compression_opts.window_bits: -14
   -66> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                  Options.compression_opts.level: 32767
   -65> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:               Options.compression_opts.strategy: 0
   -64> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
   -63> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:         Options.compression_opts.zstd_max_train_bytes: 0
   -62> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                  Options.compression_opts.enabled: false
   -61> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
   -60> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
   -59> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:              Options.level0_stop_writes_trigger: 36
   -58> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                   Options.target_file_size_base: 67108864
   -57> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:             Options.target_file_size_multiplier: 1
   -56> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
   -55> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
   -54> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
   -53> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
   -52> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
   -51> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
   -50> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
   -49> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
   -48> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
   -47> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
   -46> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
   -45> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
   -44> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                        Options.arena_block_size: 33554432
   -43> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
   -42> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
   -41> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
   -40> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.disable_auto_compactions: 0
   -39> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                        Options.compaction_style: kCompactionStyleLevel
   -38> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                          Options.compaction_pri: kMinOverlappingRatio
   -37> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.size_ratio: 1
   -36> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
   -35> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
   -34> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
   -33> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
   -32> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
   -31> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
   -30> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
   -29> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                   Options.table_properties_collectors:
   -28> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                   Options.inplace_update_support: 0
   -27> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                 Options.inplace_update_num_locks: 10000
   -26> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
   -25> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:               Options.memtable_whole_key_filtering: 0
   -24> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:   Options.memtable_huge_page_size: 0
   -23> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                           Options.bloom_locality: 0
   -22> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                    Options.max_successive_merges: 0
   -21> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.optimize_filters_for_hits: 0
   -20> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.paranoid_file_checks: 0
   -19> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.force_consistency_checks: 0
   -18> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                Options.report_bg_io_stats: 0
   -17> 2021-10-05T20:31:28.368+0700 7f310cce5f00  4 rocksdb:                               Options.ttl: 0
   -16> 2021-10-05T20:31:28.484+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 247222 --- NotFound:

   -15> 2021-10-05T20:31:28.484+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 251966 --- NotFound:

   -14> 2021-10-05T20:31:28.484+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 247508 --- NotFound:

   -13> 2021-10-05T20:31:28.484+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 252237 --- NotFound:

   -12> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 249610 --- NotFound:

   -11> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 251798 --- NotFound:

   -10> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 251799 --- NotFound:

    -9> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 252235 --- NotFound:

    -8> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 252236 --- NotFound:

    -7> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 244769 --- NotFound:

    -6> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 242684 --- NotFound:

    -5> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 241854 --- NotFound:

    -4> 2021-10-05T20:31:28.486+0700 7f310cce5f00  2 rocksdb: [db/version_set.cc:1362] Unable to load table properties for file 241191 --- NotFound:

    -3> 2021-10-05T20:31:28.492+0700 7f310cce5f00  4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:db/MANIFEST-241072 succeeded,manifest_file_number is 241072, next_file_number is 252389, last_sequence is 5847989279, log_number is 252336,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

    -2> 2021-10-05T20:31:28.492+0700 7f310cce5f00  4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 252336

    -1> 2021-10-05T20:31:28.501+0700 7f310cce5f00  4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work
     0> 2021-10-05T20:31:28.512+0700 7f310cce5f00 -1 *** Caught signal (Aborted) **
 in thread 7f310cce5f00 thread_name:ceph-osd

If I have to reweight all the osds that would take months to finish which is not really possible :(