Actions
Bug #18097
closedosd fails after upgrade to kraken (v11.0.2) when using bluestore
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
kraken
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hi.
This week we have upgraded Ceph on our test cluster to the latest Kraken release (v11.0.2). After the upgrade our OSDs weren't able to start with the following output:
# /usr/bin/ceph-osd -f --cluster cephsml --id 81 --setuser ceph --setgroup ceph 2016-12-01 13:00:33.631065 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.631395 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.631522 7f968efeb800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. starting osd.81 at :/0 osd_data /var/lib/ceph/osd/cephsml-81 /var/lib/ceph/osd/cephsml-81/journal 2016-12-01 13:00:33.666652 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.688684 7f968efeb800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-12-01 13:00:33.962680 7f968efeb800 -1 osd.81 0 OSD::init() : unable to read osd superblock 2016-12-01 13:00:34.183917 7f968efeb800 -1 ** ERROR: osd init failed: (22) Invalid argument
When I tried to strace the failing OSD the only place where I found -EINVAL was this:
mincore(0x7ffdf492e267, 1, 0x7ffdf492e267) = -1 EINVAL (Invalid argument)
Here's a snippet of the log from our Jewel OSDs for comparison:
2016-12-01 12:37:38.890479 7f0ac63f0800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 12:37:38.890485 7f0ac63f0800 0 set uid:gid to 167:167 (ceph:ceph)
2016-12-01 12:37:38.890497 7f0ac63f0800 0 ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process ceph-osd, pid 1071216
2016-12-01 12:37:38.890536 7f0ac63f0800 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-12-01 12:37:38.944457 7f0ac63f0800 0 pidfile_write: ignore empty --pid-file
2016-12-01 12:37:38.975942 7f0ac63f0800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 12:37:38.986221 7f0ac63f0800 1 bluestore(/var/lib/ceph/osd/cephsml-24) mount path /var/lib/ceph/osd/cephsml-24
2016-12-01 12:37:38.986295 7f0ac63f0800 1 bdev create path /var/lib/ceph/osd/cephsml-24/block type kernel
2016-12-01 12:37:38.987028 7f0ac63f0800 1 bdev(/var/lib/ceph/osd/cephsml-24/block) open path /var/lib/ceph/osd/cephsml-24/block
2016-12-01 12:37:38.987138 7f0ac63f0800 1 bdev(/var/lib/ceph/osd/cephsml-24/block) open size 1968398401536 (1833 GB) block_size 4096 (4096 B)
2016-12-01 12:37:38.990948 7f0ac63f0800 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-12-01 12:37:38.991252 7f0ac63f0800 0 set rocksdb option compression = kNoCompression
2016-12-01 12:37:38.991286 7f0ac63f0800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 12:37:38.991312 7f0ac63f0800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 12:37:38.991356 7f0ac63f0800 0 set rocksdb option compression = kNoCompression
2016-12-01 12:37:38.991384 7f0ac63f0800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 12:37:38.991410 7f0ac63f0800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 12:37:38.991523 7f0ac63f0800 4 rocksdb: RocksDB version: 4.3.0
2016-12-01 12:37:38.991549 7f0ac63f0800 4 rocksdb: Git sha rocksdb_build_git_sha:
2016-12-01 12:37:38.991567 7f0ac63f0800 4 rocksdb: Compile date Sep 21 2016
2016-12-01 12:37:38.991741 7f0ac63f0800 4 rocksdb: DB SUMMARY
2016-12-01 12:37:38.992665 7f0ac63f0800 4 rocksdb: CURRENT file: CURRENT
2016-12-01 12:37:38.992694 7f0ac63f0800 4 rocksdb: IDENTITY file: IDENTITY
2016-12-01 12:37:38.992717 7f0ac63f0800 4 rocksdb: MANIFEST file: MANIFEST-032684 size: 382618 Bytes
2016-12-01 12:37:38.992739 7f0ac63f0800 4 rocksdb: SST files in /var/lib/ceph/osd/cephsml-24/db dir, Total Num: 1353, files: 019794.sst 020521.sst 020829.sst 020986.sst 021442.sst 021479.sst 021496.sst 021498.sst 021757.sst
2016-12-01 12:37:38.992760 7f0ac63f0800 4 rocksdb: Write Ahead Log file in /var/lib/ceph/osd/cephsml-24/db: 032673.log size: 0 ; 032676.log size: 0 ; 032679.log size: 0 ; 032682.log size: 0 ; 032685.log size: 0 ;
2016-12-01 12:37:38.992831 7f0ac63f0800 4 rocksdb: Options.error_if_exists: 0
2016-12-01 12:37:38.992855 7f0ac63f0800 4 rocksdb: Options.create_if_missing: 0
2016-12-01 12:37:38.992882 7f0ac63f0800 4 rocksdb: Options.paranoid_checks: 1
2016-12-01 12:37:38.992900 7f0ac63f0800 4 rocksdb: Options.env: 0x7f0ac73dd060
2016-12-01 12:37:38.992918 7f0ac63f0800 4 rocksdb: Options.info_log: 0x7f0ad12cbec0
2016-12-01 12:37:38.992936 7f0ac63f0800 4 rocksdb: Options.max_open_files: 5000
2016-12-01 12:37:38.992954 7f0ac63f0800 4 rocksdb: Options.max_file_opening_threads: 1
2016-12-01 12:37:38.992971 7f0ac63f0800 4 rocksdb: Options.max_total_wal_size: 0
2016-12-01 12:37:38.992990 7f0ac63f0800 4 rocksdb: Options.disableDataSync: 0
2016-12-01 12:37:38.993008 7f0ac63f0800 4 rocksdb: Options.use_fsync: 0
2016-12-01 12:37:38.993036 7f0ac63f0800 4 rocksdb: Options.max_log_file_size: 0
2016-12-01 12:37:38.993057 7f0ac63f0800 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2016-12-01 12:37:38.993075 7f0ac63f0800 4 rocksdb: Options.log_file_time_to_roll: 0
2016-12-01 12:37:38.993093 7f0ac63f0800 4 rocksdb: Options.keep_log_file_num: 1000
2016-12-01 12:37:38.993113 7f0ac63f0800 4 rocksdb: Options.recycle_log_file_num: 0
2016-12-01 12:37:38.993131 7f0ac63f0800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 12:37:38.993149 7f0ac63f0800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 12:37:38.993167 7f0ac63f0800 4 rocksdb: Options.allow_fallocate: 1
2016-12-01 12:37:38.993184 7f0ac63f0800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 12:37:38.993202 7f0ac63f0800 4 rocksdb: Options.create_missing_column_families: 0
2016-12-01 12:37:38.993220 7f0ac63f0800 4 rocksdb: Options.db_log_dir:
2016-12-01 12:37:38.993238 7f0ac63f0800 4 rocksdb: Options.wal_dir: /var/lib/ceph/osd/cephsml-24/db
2016-12-01 12:37:38.993256 7f0ac63f0800 4 rocksdb: Options.table_cache_numshardbits: 4
2016-12-01 12:37:38.993274 7f0ac63f0800 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2016-12-01 12:37:38.993292 7f0ac63f0800 4 rocksdb: Options.max_background_compactions: 1
2016-12-01 12:37:38.993309 7f0ac63f0800 4 rocksdb: Options.max_subcompactions: 1
2016-12-01 12:37:38.993327 7f0ac63f0800 4 rocksdb: Options.max_background_flushes: 1
2016-12-01 12:37:38.993346 7f0ac63f0800 4 rocksdb: Options.WAL_ttl_seconds: 0
2016-12-01 12:37:38.993366 7f0ac63f0800 4 rocksdb: Options.WAL_size_limit_MB: 0
2016-12-01 12:37:38.993384 7f0ac63f0800 4 rocksdb: Options.manifest_preallocation_size: 4194304
2016-12-01 12:37:38.993403 7f0ac63f0800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 12:37:38.993421 7f0ac63f0800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 12:37:38.993438 7f0ac63f0800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 12:37:38.993456 7f0ac63f0800 4 rocksdb: Options.is_fd_close_on_exec: 1
2016-12-01 12:37:38.993473 7f0ac63f0800 4 rocksdb: Options.stats_dump_period_sec: 600
2016-12-01 12:37:38.993491 7f0ac63f0800 4 rocksdb: Options.advise_random_on_open: 1
2016-12-01 12:37:38.993508 7f0ac63f0800 4 rocksdb: Options.db_write_buffer_size: 0d
2016-12-01 12:37:38.993528 7f0ac63f0800 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
2016-12-01 12:37:38.993546 7f0ac63f0800 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2016-12-01 12:37:38.993564 7f0ac63f0800 4 rocksdb: Options.compaction_readahead_size: 0d
2016-12-01 12:37:38.993582 7f0ac63f0800 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
2016-12-01 12:37:38.993600 7f0ac63f0800 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
2016-12-01 12:37:38.993620 7f0ac63f0800 4 rocksdb: Options.use_adaptive_mutex: 0
2016-12-01 12:37:38.993638 7f0ac63f0800 4 rocksdb: Options.rate_limiter: (nil)
2016-12-01 12:37:38.993656 7f0ac63f0800 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
2016-12-01 12:37:38.993674 7f0ac63f0800 4 rocksdb: Options.bytes_per_sync: 0
2016-12-01 12:37:38.993692 7f0ac63f0800 4 rocksdb: Options.wal_bytes_per_sync: 0
2016-12-01 12:37:38.993709 7f0ac63f0800 4 rocksdb: Options.wal_recovery_mode: 0
2016-12-01 12:37:38.993726 7f0ac63f0800 4 rocksdb: Options.enable_thread_tracking: 0
2016-12-01 12:37:38.993745 7f0ac63f0800 4 rocksdb: Options.row_cache: None
2016-12-01 12:37:38.993770 7f0ac63f0800 4 rocksdb: Options.wal_filter: None
2016-12-01 12:37:38.993788 7f0ac63f0800 4 rocksdb: Compression algorithms supported:
2016-12-01 12:37:38.993807 7f0ac63f0800 4 rocksdb: Snappy supported: 1
2016-12-01 12:37:38.993825 7f0ac63f0800 4 rocksdb: Zlib supported: 1
2016-12-01 12:37:38.993843 7f0ac63f0800 4 rocksdb: Bzip supported: 0
2016-12-01 12:37:38.993866 7f0ac63f0800 4 rocksdb: LZ4 supported: 0
2016-12-01 12:37:38.993887 7f0ac63f0800 4 rocksdb: Fast CRC32 supported: 0
2016-12-01 12:37:38.993985 7f0ac63f0800 4 rocksdb: Recovering from manifest file: MANIFEST-032684
2016-12-01 12:37:38.994060 7f0ac63f0800 4 rocksdb: --------------- Options for column family [default]:
2016-12-01 12:37:38.994089 7f0ac63f0800 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2016-12-01 12:37:38.994111 7f0ac63f0800 4 rocksdb: Options.merge_operator: None
2016-12-01 12:37:38.994129 7f0ac63f0800 4 rocksdb: Options.compaction_filter: None
2016-12-01 12:37:38.994147 7f0ac63f0800 4 rocksdb: Options.compaction_filter_factory: None
2016-12-01 12:37:38.994165 7f0ac63f0800 4 rocksdb: Options.memtable_factory: SkipListFactory
2016-12-01 12:37:38.994182 7f0ac63f0800 4 rocksdb: Options.table_factory: BlockBasedTable
2016-12-01 12:37:38.994217 7f0ac63f0800 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f0ad11ee0e8)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7f0ad12238d8
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0
2016-12-01 12:37:38.994241 7f0ac63f0800 4 rocksdb: Options.write_buffer_size: 4194304
2016-12-01 12:37:38.994260 7f0ac63f0800 4 rocksdb: Options.max_write_buffer_number: 16
2016-12-01 12:37:38.994279 7f0ac63f0800 4 rocksdb: Options.compression: NoCompression
2016-12-01 12:37:38.994296 7f0ac63f0800 4 rocksdb: Options.prefix_extractor: nullptr
2016-12-01 12:37:38.994315 7f0ac63f0800 4 rocksdb: Options.num_levels: 7
2016-12-01 12:37:38.994334 7f0ac63f0800 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
2016-12-01 12:37:38.994354 7f0ac63f0800 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2016-12-01 12:37:38.994372 7f0ac63f0800 4 rocksdb: Options.compression_opts.window_bits: -14
2016-12-01 12:37:38.994389 7f0ac63f0800 4 rocksdb: Options.compression_opts.level: -1
2016-12-01 12:37:38.994406 7f0ac63f0800 4 rocksdb: Options.compression_opts.strategy: 0
2016-12-01 12:37:38.994423 7f0ac63f0800 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2016-12-01 12:37:38.994440 7f0ac63f0800 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2016-12-01 12:37:38.994457 7f0ac63f0800 4 rocksdb: Options.level0_stop_writes_trigger: 24
2016-12-01 12:37:38.994473 7f0ac63f0800 4 rocksdb: Options.target_file_size_base: 2097152
2016-12-01 12:37:38.994491 7f0ac63f0800 4 rocksdb: Options.target_file_size_multiplier: 1
2016-12-01 12:37:38.994509 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_base: 10485760
2016-12-01 12:37:38.994527 7f0ac63f0800 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2016-12-01 12:37:38.994544 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
2016-12-01 12:37:38.994561 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2016-12-01 12:37:38.994579 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2016-12-01 12:37:38.994596 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2016-12-01 12:37:38.994614 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2016-12-01 12:37:38.994639 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2016-12-01 12:37:38.994657 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2016-12-01 12:37:38.994675 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2016-12-01 12:37:38.994692 7f0ac63f0800 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2016-12-01 12:37:38.994709 7f0ac63f0800 4 rocksdb: Options.expanded_compaction_factor: 25
2016-12-01 12:37:38.994725 7f0ac63f0800 4 rocksdb: Options.source_compaction_factor: 1
2016-12-01 12:37:38.994742 7f0ac63f0800 4 rocksdb: Options.max_grandparent_overlap_factor: 10
2016-12-01 12:37:38.994759 7f0ac63f0800 4 rocksdb: Options.arena_block_size: 524288
2016-12-01 12:37:38.994775 7f0ac63f0800 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
2016-12-01 12:37:38.994792 7f0ac63f0800 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
2016-12-01 12:37:38.994810 7f0ac63f0800 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2016-12-01 12:37:38.994827 7f0ac63f0800 4 rocksdb: Options.disable_auto_compactions: 0
2016-12-01 12:37:38.994843 7f0ac63f0800 4 rocksdb: Options.filter_deletes: 0
2016-12-01 12:37:38.994865 7f0ac63f0800 4 rocksdb: Options.verify_checksums_in_compaction: 1
2016-12-01 12:37:38.994887 7f0ac63f0800 4 rocksdb: Options.compaction_style: 0
2016-12-01 12:37:38.994904 7f0ac63f0800 4 rocksdb: Options.compaction_pri: 0
2016-12-01 12:37:38.994922 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2016-12-01 12:37:38.994939 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2016-12-01 12:37:38.994956 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2016-12-01 12:37:38.994973 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2016-12-01 12:37:38.994991 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2016-12-01 12:37:38.995008 7f0ac63f0800 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2016-12-01 12:37:38.995025 7f0ac63f0800 4 rocksdb: Options.table_properties_collectors:
2016-12-01 12:37:38.995043 7f0ac63f0800 4 rocksdb: Options.inplace_update_support: 0
2016-12-01 12:37:38.995060 7f0ac63f0800 4 rocksdb: Options.inplace_update_num_locks: 10000
2016-12-01 12:37:38.995076 7f0ac63f0800 4 rocksdb: Options.min_partial_merge_operands: 2
2016-12-01 12:37:38.995093 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
2016-12-01 12:37:38.995111 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
2016-12-01 12:37:38.995130 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
2016-12-01 12:37:38.995148 7f0ac63f0800 4 rocksdb: Options.bloom_locality: 0
2016-12-01 12:37:38.995165 7f0ac63f0800 4 rocksdb: Options.max_successive_merges: 0
2016-12-01 12:37:38.995183 7f0ac63f0800 4 rocksdb: Options.optimize_fllters_for_hits: 0
2016-12-01 12:37:38.995200 7f0ac63f0800 4 rocksdb: Options.paranoid_file_checks: 0
2016-12-01 12:37:38.995217 7f0ac63f0800 4 rocksdb: Options.compaction_measure_io_stats: 0
2016-12-01 12:37:39.007146 7f0ac63f0800 4 rocksdb: Recovered from manifest file:/var/lib/ceph/osd/cephsml-24/db/MANIFEST-032684 succeeded,manifest_file_number is 32684, next_file_number is 32686, last_sequence is 29816718, log_number is 0,prev_log_number is 0,max_column_family is 0
2016-12-01 12:37:39.007161 7f0ac63f0800 4 rocksdb: Column family [default] (ID 0), log number is 32671
2016-12-01 12:37:39.012126 7f0ac63f0800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480585059012118, "job": 1, "event": "recovery_started", "log_files": [32673, 32676, 32679, 32682, 32685]}
2016-12-01 12:37:39.012152 7f0ac63f0800 4 rocksdb: Recovering log #32673 mode 0 skip-recovery 0
2016-12-01 12:37:39.012181 7f0ac63f0800 4 rocksdb: Recovering log #32676 mode 0 skip-recovery 0
2016-12-01 12:37:39.012198 7f0ac63f0800 4 rocksdb: Recovering log #32679 mode 0 skip-recovery 0
2016-12-01 12:37:39.012213 7f0ac63f0800 4 rocksdb: Recovering log #32682 mode 0 skip-recovery 0
2016-12-01 12:37:39.012228 7f0ac63f0800 4 rocksdb: Recovering log #32685 mode 0 skip-recovery 0
2016-12-01 12:37:39.012290 7f0ac63f0800 4 rocksdb: Creating manifest 32687
2016-12-01 12:37:39.044359 7f0ac63f0800 4 rocksdb: Deleting manifest 32684 current manifest 32687
2016-12-01 12:37:39.044699 7f0ac63f0800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480585059044692, "job": 1, "event": "recovery_finished"}
Files
Updated by Yuri Gorshkov over 7 years ago
More loggs from the affected OSD:
/usr/bin/ceph-osd -f --cluster cephsml --id 81 --setuser ceph --setgroup ceph -d --debug-osd 200
2016-12-01 13:51:58.993407 7f6193274800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 13:51:58.993690 7f6193274800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 13:51:58.993700 7f6193274800 0 set uid:gid to 167:167 (ceph:ceph)
2016-12-01 13:51:58.993732 7f6193274800 0 ceph version v11.0.2 (697fe64f9f106252c49a2c4fe4d79aea29363be7), process ceph-osd, pid 3576637
2016-12-01 13:51:58.993794 7f6193274800 5 object store type is bluestore
2016-12-01 13:51:58.993830 7f6193274800 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
starting osd.81 at :/0 osd_data /var/lib/ceph/osd/cephsml-81 /var/lib/ceph/osd/cephsml-81/journal
2016-12-01 13:51:58.996401 7f6193274800 0 pidfile_write: ignore empty --pid-file
2016-12-01 13:51:59.029845 7f6193274800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 13:51:59.039350 7f6193274800 2 osd.81 0 mounting /var/lib/ceph/osd/cephsml-81 /var/lib/ceph/osd/cephsml-81/journal
2016-12-01 13:51:59.039364 7f6193274800 1 bluestore(/var/lib/ceph/osd/cephsml-81) mount path /var/lib/ceph/osd/cephsml-81
2016-12-01 13:51:59.039411 7f6193274800 1 bdev create path /var/lib/ceph/osd/cephsml-81/block type kernel
2016-12-01 13:51:59.040129 7f6193274800 1 bdev(/var/lib/ceph/osd/cephsml-81/block) open path /var/lib/ceph/osd/cephsml-81/block
2016-12-01 13:51:59.040583 7f6193274800 1 bdev(/var/lib/ceph/osd/cephsml-81/block) open size 1968398401536 (0x1ca4db00000, 1833 GB) block_size 4096 (4096 B) non-rotational
2016-12-01 13:51:59.040842 7f6193274800 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-12-01 13:51:59.040964 7f6193274800 0 set rocksdb option compression = kNoCompression
2016-12-01 13:51:59.040976 7f6193274800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 13:51:59.040981 7f6193274800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 13:51:59.041022 7f6193274800 0 set rocksdb option compression = kNoCompression
2016-12-01 13:51:59.041031 7f6193274800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 13:51:59.041036 7f6193274800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 13:51:59.041202 7f6193274800 4 rocksdb: RocksDB version: 4.12.0
2016-12-01 13:51:59.041225 7f6193274800 4 rocksdb: Git sha rocksdb_build_git_sha:
2016-12-01 13:51:59.041227 7f6193274800 4 rocksdb: Compile date Oct 17 2016
2016-12-01 13:51:59.041230 7f6193274800 4 rocksdb: DB SUMMARY
2016-12-01 13:51:59.042240 7f6193274800 4 rocksdb: CURRENT file: CURRENT
2016-12-01 13:51:59.042247 7f6193274800 4 rocksdb: IDENTITY file: IDENTITY
2016-12-01 13:51:59.042251 7f6193274800 4 rocksdb: MANIFEST file: MANIFEST-034277 size: 419298 Bytes
2016-12-01 13:51:59.042254 7f6193274800 4 rocksdb: SST files in /var/lib/ceph/osd/cephsml-81/db dir, Total Num: 1477, files: 018890.sst 018942.sst 018981.sst 019590.sst 019618.sst 020538.sst 020779.sst 020830.sst 020992.sst
2016-12-01 13:51:59.042258 7f6193274800 4 rocksdb: Write Ahead Log file in /var/lib/ceph/osd/cephsml-81/db: 034242.log size: 0 ; 034248.log size: 0 ; 034251.log size: 0 ; 034254.log size: 0 ; 034257.log size: 0 ; 034260.log size: 0 ; 034263.log size: 0 ; 034266.log size: 0 ; 034269.log size: 0 ; 034272.log size: 0 ; 034275.log size: 0 ; 034278.log size: 0 ;
2016-12-01 13:51:59.042325 7f6193274800 4 rocksdb: Options.error_if_exists: 0
2016-12-01 13:51:59.042327 7f6193274800 4 rocksdb: Options.create_if_missing: 0
2016-12-01 13:51:59.042328 7f6193274800 4 rocksdb: Options.paranoid_checks: 1
2016-12-01 13:51:59.042329 7f6193274800 4 rocksdb: Options.env: 0x7f619438a740
2016-12-01 13:51:59.042330 7f6193274800 4 rocksdb: Options.info_log: 0x7f619ef5d120
2016-12-01 13:51:59.042332 7f6193274800 4 rocksdb: Options.max_open_files: -1
2016-12-01 13:51:59.042332 7f6193274800 4 rocksdb: Options.max_file_opening_threads: 16
2016-12-01 13:51:59.042333 7f6193274800 4 rocksdb: Options.max_total_wal_size: 0
2016-12-01 13:51:59.042334 7f6193274800 4 rocksdb: Options.disableDataSync: 0
2016-12-01 13:51:59.042335 7f6193274800 4 rocksdb: Options.use_fsync: 0
2016-12-01 13:51:59.042336 7f6193274800 4 rocksdb: Options.max_log_file_size: 0
2016-12-01 13:51:59.042337 7f6193274800 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2016-12-01 13:51:59.042340 7f6193274800 4 rocksdb: Options.log_file_time_to_roll: 0
2016-12-01 13:51:59.042343 7f6193274800 4 rocksdb: Options.keep_log_file_num: 1000
2016-12-01 13:51:59.042344 7f6193274800 4 rocksdb: Options.recycle_log_file_num: 0
2016-12-01 13:51:59.042345 7f6193274800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 13:51:59.042346 7f6193274800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 13:51:59.042347 7f6193274800 4 rocksdb: Options.allow_fallocate: 1
2016-12-01 13:51:59.042349 7f6193274800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 13:51:59.042351 7f6193274800 4 rocksdb: Options.create_missing_column_families: 0
2016-12-01 13:51:59.042351 7f6193274800 4 rocksdb: Options.db_log_dir:
2016-12-01 13:51:59.042352 7f6193274800 4 rocksdb: Options.wal_dir: /var/lib/ceph/osd/cephsml-81/db
2016-12-01 13:51:59.042354 7f6193274800 4 rocksdb: Options.table_cache_numshardbits: 6
2016-12-01 13:51:59.042354 7f6193274800 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2016-12-01 13:51:59.042355 7f6193274800 4 rocksdb: Options.base_background_compactions: 1
2016-12-01 13:51:59.042356 7f6193274800 4 rocksdb: Options.max_background_compactions: 1
2016-12-01 13:51:59.042357 7f6193274800 4 rocksdb: Options.max_subcompactions: 1
2016-12-01 13:51:59.042358 7f6193274800 4 rocksdb: Options.max_background_flushes: 1
2016-12-01 13:51:59.042365 7f6193274800 4 rocksdb: Options.WAL_ttl_seconds: 0
2016-12-01 13:51:59.042366 7f6193274800 4 rocksdb: Options.WAL_size_limit_MB: 0
2016-12-01 13:51:59.042367 7f6193274800 4 rocksdb: Options.manifest_preallocation_size: 4194304
2016-12-01 13:51:59.042368 7f6193274800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 13:51:59.042368 7f6193274800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 13:51:59.042369 7f6193274800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 13:51:59.042370 7f6193274800 4 rocksdb: Options.is_fd_close_on_exec: 1
2016-12-01 13:51:59.042371 7f6193274800 4 rocksdb: Options.stats_dump_period_sec: 600
2016-12-01 13:51:59.042372 7f6193274800 4 rocksdb: Options.advise_random_on_open: 1
2016-12-01 13:51:59.042373 7f6193274800 4 rocksdb: Options.db_write_buffer_size: 0d
2016-12-01 13:51:59.042374 7f6193274800 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
2016-12-01 13:51:59.042375 7f6193274800 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2016-12-01 13:51:59.042376 7f6193274800 4 rocksdb: Options.compaction_readahead_size: 0d
2016-12-01 13:51:59.042376 7f6193274800 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
2016-12-01 13:51:59.042377 7f6193274800 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
2016-12-01 13:51:59.042378 7f6193274800 4 rocksdb: Options.use_adaptive_mutex: 0
2016-12-01 13:51:59.042379 7f6193274800 4 rocksdb: Options.rate_limiter: (nil)
2016-12-01 13:51:59.042381 7f6193274800 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2016-12-01 13:51:59.042382 7f6193274800 4 rocksdb: Options.bytes_per_sync: 0
2016-12-01 13:51:59.042383 7f6193274800 4 rocksdb: Options.wal_bytes_per_sync: 0
2016-12-01 13:51:59.042384 7f6193274800 4 rocksdb: Options.wal_recovery_mode: 2
2016-12-01 13:51:59.042385 7f6193274800 4 rocksdb: Options.enable_thread_tracking: 0
2016-12-01 13:51:59.042386 7f6193274800 4 rocksdb: Options.allow_concurrent_memtable_write: 0
2016-12-01 13:51:59.042386 7f6193274800 4 rocksdb: Options.enable_write_thread_adaptive_yield: 0
2016-12-01 13:51:59.042387 7f6193274800 4 rocksdb: Options.write_thread_max_yield_usec: 100
2016-12-01 13:51:59.042388 7f6193274800 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2016-12-01 13:51:59.042389 7f6193274800 4 rocksdb: Options.row_cache: None
2016-12-01 13:51:59.042390 7f6193274800 4 rocksdb: Options.wal_filter: None
2016-12-01 13:51:59.042391 7f6193274800 4 rocksdb: Options.avoid_flush_during_recovery: 0
2016-12-01 13:51:59.042392 7f6193274800 4 rocksdb: Compression algorithms supported:
2016-12-01 13:51:59.042393 7f6193274800 4 rocksdb: Snappy supported: 1
2016-12-01 13:51:59.042393 7f6193274800 4 rocksdb: Zlib supported: 1
2016-12-01 13:51:59.042394 7f6193274800 4 rocksdb: Bzip supported: 0
2016-12-01 13:51:59.042395 7f6193274800 4 rocksdb: LZ4 supported: 0
2016-12-01 13:51:59.042396 7f6193274800 4 rocksdb: Fast CRC32 supported: 0
2016-12-01 13:51:59.042497 7f6193274800 4 rocksdb: Recovering from manifest file: MANIFEST-034277
2016-12-01 13:51:59.042562 7f6193274800 4 rocksdb: --------------- Options for column family [default]:
2016-12-01 13:51:59.042578 7f6193274800 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2016-12-01 13:51:59.042588 7f6193274800 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor
2016-12-01 13:51:59.042589 7f6193274800 4 rocksdb: Options.compaction_filter: None
2016-12-01 13:51:59.042590 7f6193274800 4 rocksdb: Options.compaction_filter_factory: None
2016-12-01 13:51:59.042591 7f6193274800 4 rocksdb: Options.memtable_factory: SkipListFactory
2016-12-01 13:51:59.042592 7f6193274800 4 rocksdb: Options.table_factory: BlockBasedTable
2016-12-01 13:51:59.042610 7f6193274800 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f619ed60150)
cache_index_and_filter_blocks: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7f619ed9a2a8
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 2
2016-12-01 13:51:59.042614 7f6193274800 4 rocksdb: Options.write_buffer_size: 67108864
2016-12-01 13:51:59.042616 7f6193274800 4 rocksdb: Options.max_write_buffer_number: 16
2016-12-01 13:51:59.042617 7f6193274800 4 rocksdb: Options.compression: NoCompression
2016-12-01 13:51:59.042618 7f6193274800 4 rocksdb: Options.bottommost_compression: Disabled
2016-12-01 13:51:59.042619 7f6193274800 4 rocksdb: Options.prefix_extractor: nullptr
2016-12-01 13:51:59.042620 7f6193274800 4 rocksdb: Options.num_levels: 7
2016-12-01 13:51:59.042621 7f6193274800 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
2016-12-01 13:51:59.042622 7f6193274800 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2016-12-01 13:51:59.042622 7f6193274800 4 rocksdb: Options.compression_opts.window_bits: -14
2016-12-01 13:51:59.042624 7f6193274800 4 rocksdb: Options.compression_opts.level: -1
2016-12-01 13:51:59.042625 7f6193274800 4 rocksdb: Options.compression_opts.strategy: 0
2016-12-01 13:51:59.042626 7f6193274800 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2016-12-01 13:51:59.042627 7f6193274800 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2016-12-01 13:51:59.042628 7f6193274800 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2016-12-01 13:51:59.042629 7f6193274800 4 rocksdb: Options.level0_stop_writes_trigger: 24
2016-12-01 13:51:59.042630 7f6193274800 4 rocksdb: Options.target_file_size_base: 67108864
2016-12-01 13:51:59.042630 7f6193274800 4 rocksdb: Options.target_file_size_multiplier: 1
2016-12-01 13:51:59.042631 7f6193274800 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2016-12-01 13:51:59.042632 7f6193274800 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2016-12-01 13:51:59.042633 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
2016-12-01 13:51:59.042634 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2016-12-01 13:51:59.042635 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2016-12-01 13:51:59.042636 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2016-12-01 13:51:59.042637 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2016-12-01 13:51:59.042638 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2016-12-01 13:51:59.042639 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2016-12-01 13:51:59.042640 7f6193274800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2016-12-01 13:51:59.042641 7f6193274800 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2016-12-01 13:51:59.042641 7f6193274800 4 rocksdb: Options.max_compaction_bytes: 1677721600
2016-12-01 13:51:59.042643 7f6193274800 4 rocksdb: Options.arena_block_size: 8388608
2016-12-01 13:51:59.042644 7f6193274800 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2016-12-01 13:51:59.042645 7f6193274800 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2016-12-01 13:51:59.042646 7f6193274800 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2016-12-01 13:51:59.042647 7f6193274800 4 rocksdb: Options.disable_auto_compactions: 0
2016-12-01 13:51:59.042647 7f6193274800 4 rocksdb: Options.verify_checksums_in_compaction: 1
2016-12-01 13:51:59.042648 7f6193274800 4 rocksdb: Options.compaction_style: 0
2016-12-01 13:51:59.042649 7f6193274800 4 rocksdb: Options.compaction_pri: 0
2016-12-01 13:51:59.042650 7f6193274800 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2016-12-01 13:51:59.042651 7f6193274800 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2016-12-01 13:51:59.042653 7f6193274800 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2016-12-01 13:51:59.042654 7f6193274800 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2016-12-01 13:51:59.042655 7f6193274800 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2016-12-01 13:51:59.042656 7f6193274800 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2016-12-01 13:51:59.042657 7f6193274800 4 rocksdb: Options.table_properties_collectors:
2016-12-01 13:51:59.042658 7f6193274800 4 rocksdb: Options.inplace_update_support: 0
2016-12-01 13:51:59.042659 7f6193274800 4 rocksdb: Options.inplace_update_num_locks: 10000
2016-12-01 13:51:59.042659 7f6193274800 4 rocksdb: Options.min_partial_merge_operands: 2
2016-12-01 13:51:59.042660 7f6193274800 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2016-12-01 13:51:59.042673 7f6193274800 4 rocksdb: Options.memtable_huge_page_size: 0
2016-12-01 13:51:59.042674 7f6193274800 4 rocksdb: Options.bloom_locality: 0
2016-12-01 13:51:59.042675 7f6193274800 4 rocksdb: Options.max_successive_merges: 0
2016-12-01 13:51:59.042676 7f6193274800 4 rocksdb: Options.optimize_filters_for_hits: 0
2016-12-01 13:51:59.042677 7f6193274800 4 rocksdb: Options.paranoid_file_checks: 0
2016-12-01 13:51:59.042678 7f6193274800 4 rocksdb: Options.report_bg_io_stats: 0
2016-12-01 13:51:59.064427 7f6193274800 4 rocksdb: Recovered from manifest file:/var/lib/ceph/osd/cephsml-81/db/MANIFEST-034277 succeeded,manifest_file_number is 34277, next_file_number is 34279, last_sequence is 28658480, log_number is 0,prev_log_number is 0,max_column_family is 0
2016-12-01 13:51:59.064448 7f6193274800 4 rocksdb: Column family [default] (ID 0), log number is 34240
2016-12-01 13:51:59.068080 7f6193274800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480589519068069, "job": 1, "event": "recovery_started", "log_files": [34242, 34248, 34251, 34254, 34257, 34260, 34263, 34266, 34269, 34272, 34275, 34278]}
2016-12-01 13:51:59.068089 7f6193274800 4 rocksdb: Recovering log #34242 mode 2
2016-12-01 13:51:59.068151 7f6193274800 4 rocksdb: Recovering log #34248 mode 2
2016-12-01 13:51:59.068168 7f6193274800 4 rocksdb: Recovering log #34251 mode 2
2016-12-01 13:51:59.068181 7f6193274800 4 rocksdb: Recovering log #34254 mode 2
2016-12-01 13:51:59.068191 7f6193274800 4 rocksdb: Recovering log #34257 mode 2
2016-12-01 13:51:59.068200 7f6193274800 4 rocksdb: Recovering log #34260 mode 2
2016-12-01 13:51:59.068210 7f6193274800 4 rocksdb: Recovering log #34263 mode 2
2016-12-01 13:51:59.068221 7f6193274800 4 rocksdb: Recovering log #34266 mode 2
2016-12-01 13:51:59.068231 7f6193274800 4 rocksdb: Recovering log #34269 mode 2
2016-12-01 13:51:59.068245 7f6193274800 4 rocksdb: Recovering log #34272 mode 2
2016-12-01 13:51:59.068255 7f6193274800 4 rocksdb: Recovering log #34275 mode 2
2016-12-01 13:51:59.068264 7f6193274800 4 rocksdb: Recovering log #34278 mode 2
2016-12-01 13:51:59.068317 7f6193274800 4 rocksdb: Creating manifest 34280
2016-12-01 13:51:59.098921 7f6193274800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480589519098916, "job": 1, "event": "recovery_finished"}
2016-12-01 13:51:59.124641 7f6193274800 4 rocksdb: DB pointer 0x7f619ee8aa00
2016-12-01 13:51:59.124687 7f6193274800 1 bluestore(/var/lib/ceph/osd/cephsml-81) _open_db opened rocksdb path /var/lib/ceph/osd/cephsml-81/db options compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3
2016-12-01 13:51:59.124881 7f6193274800 1 freelist init
2016-12-01 13:51:59.313322 7f6193274800 2 osd.81 0 boot
2016-12-01 13:51:59.313397 7f6193274800 20 osd.81 0 configured osd_max_object_name[space]_len looks ok
2016-12-01 13:51:59.313455 7f6193274800 -1 osd.81 0 OSD::init() : unable to read osd superblock
2016-12-01 13:51:59.313463 7f6193274800 1 bluestore(/var/lib/ceph/osd/cephsml-81) umount
2016-12-01 13:51:59.396738 7f6193274800 1 freelist shutdown
2016-12-01 13:51:59.414811 7f6193274800 1 bdev(/var/lib/ceph/osd/cephsml-81/block) close
2016-12-01 13:51:59.543805 7f6193274800 -1 ** ERROR: osd init failed: (22) Invalid argument
Updated by Yuri Gorshkov over 7 years ago
- File mk_osd_bluestore.sh mk_osd_bluestore.sh added
Forgot to mention perhaps the most important part: due to hitting a lot of various bluefs bugs in Jewel we were using Bluestore without bluefs (e.g. with RocksDB residing on XFS).
The script for making such an arrangement is attached below.
When I reinitialized a Kraken OSD in the same way using this script it was able to get up and running without issues. Old Jewel OSDs that were using the same arrangement are all failing to start up with the above output.
Also, I'm attaching the relevant sections of our ceph.conf:
[osd] osd mkfs type = xfs osd mkfs options xfs = -f -i size=2048 osd mount options xfs = noatime,largeio,inode64,swalloc osd journal size = 2600 osd op threads = 4 osd recovery delay start = 15 osd recovery threads = 2 osd objectstore = bluestore bluestore block path = /dev/disk/by-partlabel/osd-device-$id-block bluestore bluefs = false bluestore fsck on mount = false bluestore block db path = /var/lib/ceph/osd/$cluster-$id/block.db bluestore block db create = true bluestore rocksdb options = compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3
Please note that as we've decided to proceed with reinitialize the whole cluster from scratch with Kraken, we won't be able to provide more info regarding this issue - but I still think this might still be interesting to people so I'm trying to provide as much info as I can.
Updated by Sage Weil about 7 years ago
this si expected; there was an ondisk format change right before kraken was released.
Actions