Ceph » RADOS

i create a vstart.sh cluster using mimic branch, and ceph-monstore-tool from master is able to open it just fine.

$ bin/ceph-monstore-tool ~/dev/ceph/build/dev/mon.a/ dump-keys
...
2018-08-01 14:27:22.935 7f9e16c6df80  1 rocksdb: do_open column families: [default]
2018-08-01 14:27:22.935 7f9e16c6df80  4 rocksdb: RocksDB version: 5.14.0

2018-08-01 14:27:22.935 7f9e16c6df80  4 rocksdb: Git sha rocksdb_build_git_sha:@9090ae3ecfbf9b50a398a5d8b178f14b88dc047e@
2018-08-01 14:27:22.935 7f9e16c6df80  4 rocksdb: Compile date Jul 27 2018
...
2018-08-01 14:27:22.935 7f9e16c6df80  4 rocksdb: SST files in /home/kefu/dev/ceph/build/dev/mon.a/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000
013.sst
...
2018-08-01 14:27:23.067 7f9e16c6df80  4 rocksdb: [/var/ssd/ceph/src/rocksdb/db/db_impl_open.cc:1219] DB pointer 0x559081bc1000
auth / 1
auth / 2
auth / 3
auth / 4
auth / 5
auth / 6
auth / 7
auth / 8
auth / first_committed
auth / format_version
...

rerunning the test at http://pulpito.ceph.com/kchai-2018-08-01_06:21:53-upgrade:mimic-x:parallel-master-distro-basic-smithi/

it's a regression in rocksdb. the rocksdb in mimic (eaee6d3beab3429232ceb188377a3f94e844fca7) is f4a857da0b720691effc524469f6db895ad00d8e, which contains https://github.com/facebook/rocksdb/commit/73f21a7b2177aeb82b9f518222e2b9ea8fbb7c4f. this commit is fine per se. but older rocksdb not containing this commit will not be able to open the db file created by this change.

because 73f21a7b2177aeb82b9f518222e2b9ea8fbb7c4f creates dummy entry in the manifest to record the deleted WALs, expecting that the recovery will skip them. so this change is not forward compatible. i.e. old rocksdb is not necessarily able to open store created by new rocksdb.

the dummy entry looks like:

$20 = {fd = {table_reader = 0x0, packed_number_and_path_id = 0, file_size = 0}, smallest = {rep_ = "dummy_key\001\000\000\000\000\000\000"},
  largest = {rep_ = "dummy_key\001\000\000\000\000\000\000"}, smallest_seqno = 0, largest_seqno = 0, table_reader_handle = 0x0, stats = {
    num_reads_sampled = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}},
  compensated_file_size = 0, num_entries = 0, num_deletions = 0, raw_key_size = 0, raw_value_size = 0, refs = 1, being_compacted = false,
  init_stats_from_file = false, marked_for_compaction = false}

but when rocksdb identified that this change is not forward compatible, they decided to revert this change in https://github.com/facebook/rocksdb/pull/3762. so, this change makes rocksdb not backward compatible. in other words, new rocksdb is not able to open store created by old rocksdb.

and we do have this change in master! that's why we are suffering from a fix from rocksdb upstream.

this is embarrassing. because mimic is released. i don't really want our user to rebuild their OSDs or monitors to be forward compatible with master. we probably have to bite the bullet to keep the reverted change in our fork and maintain it, unless we can persuade the upstream to revert https://github.com/facebook/rocksdb/pull/3762 .

an alternative option is to whip up a tool to rebuild the manifest to remove the dummy File4 with kDeletedLogNumberHack custom_tag. see also rocksdb/db/repair.cc .

another option would be to only partially revert, and keep just the bits that ignore the older deleted log files.

https://github.com/ceph/ceph/pull/25070