Actions
Bug #18096
closedosd fails after upgrade to kraken (v11.0.2) when using bluestore
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hi.
This week we have upgraded Ceph on our test cluster to the latest Kraken release (v11.0.2). After the upgrade our OSDs weren't able to start with the following output:
# /usr/bin/ceph-osd -f --cluster cephsml --id 81 --setuser ceph --setgroup ceph 2016-12-01 13:00:33.631065 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.631395 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.631522 7f968efeb800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. starting osd.81 at :/0 osd_data /var/lib/ceph/osd/cephsml-81 /var/lib/ceph/osd/cephsml-81/journal 2016-12-01 13:00:33.666652 7f968efeb800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-12-01 13:00:33.688684 7f968efeb800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-12-01 13:00:33.962680 7f968efeb800 -1 osd.81 0 OSD::init() : unable to read osd superblock 2016-12-01 13:00:34.183917 7f968efeb800 -1 ** ERROR: osd init failed: (22) Invalid argument
When I tried to strace the failing OSD the only place where I found -EINVAL was this:
mincore(0x7ffdf492e267, 1, 0x7ffdf492e267) = -1 EINVAL (Invalid argument)
Here's a snippet of the log from our Jewel OSDs for comparison:
2016-12-01 12:37:38.890479 7f0ac63f0800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 12:37:38.890485 7f0ac63f0800 0 set uid:gid to 167:167 (ceph:ceph)
2016-12-01 12:37:38.890497 7f0ac63f0800 0 ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process ceph-osd, pid 1071216
2016-12-01 12:37:38.890536 7f0ac63f0800 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-12-01 12:37:38.944457 7f0ac63f0800 0 pidfile_write: ignore empty --pid-file
2016-12-01 12:37:38.975942 7f0ac63f0800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-12-01 12:37:38.986221 7f0ac63f0800 1 bluestore(/var/lib/ceph/osd/cephsml-24) mount path /var/lib/ceph/osd/cephsml-24
2016-12-01 12:37:38.986295 7f0ac63f0800 1 bdev create path /var/lib/ceph/osd/cephsml-24/block type kernel
2016-12-01 12:37:38.987028 7f0ac63f0800 1 bdev(/var/lib/ceph/osd/cephsml-24/block) open path /var/lib/ceph/osd/cephsml-24/block
2016-12-01 12:37:38.987138 7f0ac63f0800 1 bdev(/var/lib/ceph/osd/cephsml-24/block) open size 1968398401536 (1833 GB) block_size 4096 (4096 B)
2016-12-01 12:37:38.990948 7f0ac63f0800 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-12-01 12:37:38.991252 7f0ac63f0800 0 set rocksdb option compression = kNoCompression
2016-12-01 12:37:38.991286 7f0ac63f0800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 12:37:38.991312 7f0ac63f0800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 12:37:38.991356 7f0ac63f0800 0 set rocksdb option compression = kNoCompression
2016-12-01 12:37:38.991384 7f0ac63f0800 0 set rocksdb option max_write_buffer_number = 16
2016-12-01 12:37:38.991410 7f0ac63f0800 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-12-01 12:37:38.991523 7f0ac63f0800 4 rocksdb: RocksDB version: 4.3.0
2016-12-01 12:37:38.991549 7f0ac63f0800 4 rocksdb: Git sha rocksdb_build_git_sha:
2016-12-01 12:37:38.991567 7f0ac63f0800 4 rocksdb: Compile date Sep 21 2016
2016-12-01 12:37:38.991741 7f0ac63f0800 4 rocksdb: DB SUMMARY
2016-12-01 12:37:38.992665 7f0ac63f0800 4 rocksdb: CURRENT file: CURRENT
2016-12-01 12:37:38.992694 7f0ac63f0800 4 rocksdb: IDENTITY file: IDENTITY
2016-12-01 12:37:38.992717 7f0ac63f0800 4 rocksdb: MANIFEST file: MANIFEST-032684 size: 382618 Bytes
2016-12-01 12:37:38.992739 7f0ac63f0800 4 rocksdb: SST files in /var/lib/ceph/osd/cephsml-24/db dir, Total Num: 1353, files: 019794.sst 020521.sst 020829.sst 020986.sst 021442.sst 021479.sst 021496.sst 021498.sst 021757.sst
2016-12-01 12:37:38.992760 7f0ac63f0800 4 rocksdb: Write Ahead Log file in /var/lib/ceph/osd/cephsml-24/db: 032673.log size: 0 ; 032676.log size: 0 ; 032679.log size: 0 ; 032682.log size: 0 ; 032685.log size: 0 ;
2016-12-01 12:37:38.992831 7f0ac63f0800 4 rocksdb: Options.error_if_exists: 0
2016-12-01 12:37:38.992855 7f0ac63f0800 4 rocksdb: Options.create_if_missing: 0
2016-12-01 12:37:38.992882 7f0ac63f0800 4 rocksdb: Options.paranoid_checks: 1
2016-12-01 12:37:38.992900 7f0ac63f0800 4 rocksdb: Options.env: 0x7f0ac73dd060
2016-12-01 12:37:38.992918 7f0ac63f0800 4 rocksdb: Options.info_log: 0x7f0ad12cbec0
2016-12-01 12:37:38.992936 7f0ac63f0800 4 rocksdb: Options.max_open_files: 5000
2016-12-01 12:37:38.992954 7f0ac63f0800 4 rocksdb: Options.max_file_opening_threads: 1
2016-12-01 12:37:38.992971 7f0ac63f0800 4 rocksdb: Options.max_total_wal_size: 0
2016-12-01 12:37:38.992990 7f0ac63f0800 4 rocksdb: Options.disableDataSync: 0
2016-12-01 12:37:38.993008 7f0ac63f0800 4 rocksdb: Options.use_fsync: 0
2016-12-01 12:37:38.993036 7f0ac63f0800 4 rocksdb: Options.max_log_file_size: 0
2016-12-01 12:37:38.993057 7f0ac63f0800 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2016-12-01 12:37:38.993075 7f0ac63f0800 4 rocksdb: Options.log_file_time_to_roll: 0
2016-12-01 12:37:38.993093 7f0ac63f0800 4 rocksdb: Options.keep_log_file_num: 1000
2016-12-01 12:37:38.993113 7f0ac63f0800 4 rocksdb: Options.recycle_log_file_num: 0
2016-12-01 12:37:38.993131 7f0ac63f0800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 12:37:38.993149 7f0ac63f0800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 12:37:38.993167 7f0ac63f0800 4 rocksdb: Options.allow_fallocate: 1
2016-12-01 12:37:38.993184 7f0ac63f0800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 12:37:38.993202 7f0ac63f0800 4 rocksdb: Options.create_missing_column_families: 0
2016-12-01 12:37:38.993220 7f0ac63f0800 4 rocksdb: Options.db_log_dir:
2016-12-01 12:37:38.993238 7f0ac63f0800 4 rocksdb: Options.wal_dir: /var/lib/ceph/osd/cephsml-24/db
2016-12-01 12:37:38.993256 7f0ac63f0800 4 rocksdb: Options.table_cache_numshardbits: 4
2016-12-01 12:37:38.993274 7f0ac63f0800 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2016-12-01 12:37:38.993292 7f0ac63f0800 4 rocksdb: Options.max_background_compactions: 1
2016-12-01 12:37:38.993309 7f0ac63f0800 4 rocksdb: Options.max_subcompactions: 1
2016-12-01 12:37:38.993327 7f0ac63f0800 4 rocksdb: Options.max_background_flushes: 1
2016-12-01 12:37:38.993346 7f0ac63f0800 4 rocksdb: Options.WAL_ttl_seconds: 0
2016-12-01 12:37:38.993366 7f0ac63f0800 4 rocksdb: Options.WAL_size_limit_MB: 0
2016-12-01 12:37:38.993384 7f0ac63f0800 4 rocksdb: Options.manifest_preallocation_size: 4194304
2016-12-01 12:37:38.993403 7f0ac63f0800 4 rocksdb: Options.allow_os_buffer: 1
2016-12-01 12:37:38.993421 7f0ac63f0800 4 rocksdb: Options.allow_mmap_reads: 0
2016-12-01 12:37:38.993438 7f0ac63f0800 4 rocksdb: Options.allow_mmap_writes: 0
2016-12-01 12:37:38.993456 7f0ac63f0800 4 rocksdb: Options.is_fd_close_on_exec: 1
2016-12-01 12:37:38.993473 7f0ac63f0800 4 rocksdb: Options.stats_dump_period_sec: 600
2016-12-01 12:37:38.993491 7f0ac63f0800 4 rocksdb: Options.advise_random_on_open: 1
2016-12-01 12:37:38.993508 7f0ac63f0800 4 rocksdb: Options.db_write_buffer_size: 0d
2016-12-01 12:37:38.993528 7f0ac63f0800 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
2016-12-01 12:37:38.993546 7f0ac63f0800 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2016-12-01 12:37:38.993564 7f0ac63f0800 4 rocksdb: Options.compaction_readahead_size: 0d
2016-12-01 12:37:38.993582 7f0ac63f0800 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
2016-12-01 12:37:38.993600 7f0ac63f0800 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
2016-12-01 12:37:38.993620 7f0ac63f0800 4 rocksdb: Options.use_adaptive_mutex: 0
2016-12-01 12:37:38.993638 7f0ac63f0800 4 rocksdb: Options.rate_limiter: (nil)
2016-12-01 12:37:38.993656 7f0ac63f0800 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
2016-12-01 12:37:38.993674 7f0ac63f0800 4 rocksdb: Options.bytes_per_sync: 0
2016-12-01 12:37:38.993692 7f0ac63f0800 4 rocksdb: Options.wal_bytes_per_sync: 0
2016-12-01 12:37:38.993709 7f0ac63f0800 4 rocksdb: Options.wal_recovery_mode: 0
2016-12-01 12:37:38.993726 7f0ac63f0800 4 rocksdb: Options.enable_thread_tracking: 0
2016-12-01 12:37:38.993745 7f0ac63f0800 4 rocksdb: Options.row_cache: None
2016-12-01 12:37:38.993770 7f0ac63f0800 4 rocksdb: Options.wal_filter: None
2016-12-01 12:37:38.993788 7f0ac63f0800 4 rocksdb: Compression algorithms supported:
2016-12-01 12:37:38.993807 7f0ac63f0800 4 rocksdb: Snappy supported: 1
2016-12-01 12:37:38.993825 7f0ac63f0800 4 rocksdb: Zlib supported: 1
2016-12-01 12:37:38.993843 7f0ac63f0800 4 rocksdb: Bzip supported: 0
2016-12-01 12:37:38.993866 7f0ac63f0800 4 rocksdb: LZ4 supported: 0
2016-12-01 12:37:38.993887 7f0ac63f0800 4 rocksdb: Fast CRC32 supported: 0
2016-12-01 12:37:38.993985 7f0ac63f0800 4 rocksdb: Recovering from manifest file: MANIFEST-032684
2016-12-01 12:37:38.994060 7f0ac63f0800 4 rocksdb: --------------- Options for column family [default]:
2016-12-01 12:37:38.994089 7f0ac63f0800 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2016-12-01 12:37:38.994111 7f0ac63f0800 4 rocksdb: Options.merge_operator: None
2016-12-01 12:37:38.994129 7f0ac63f0800 4 rocksdb: Options.compaction_filter: None
2016-12-01 12:37:38.994147 7f0ac63f0800 4 rocksdb: Options.compaction_filter_factory: None
2016-12-01 12:37:38.994165 7f0ac63f0800 4 rocksdb: Options.memtable_factory: SkipListFactory
2016-12-01 12:37:38.994182 7f0ac63f0800 4 rocksdb: Options.table_factory: BlockBasedTable
2016-12-01 12:37:38.994217 7f0ac63f0800 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f0ad11ee0e8)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7f0ad12238d8
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0
2016-12-01 12:37:38.994241 7f0ac63f0800 4 rocksdb: Options.write_buffer_size: 4194304
2016-12-01 12:37:38.994260 7f0ac63f0800 4 rocksdb: Options.max_write_buffer_number: 16
2016-12-01 12:37:38.994279 7f0ac63f0800 4 rocksdb: Options.compression: NoCompression
2016-12-01 12:37:38.994296 7f0ac63f0800 4 rocksdb: Options.prefix_extractor: nullptr
2016-12-01 12:37:38.994315 7f0ac63f0800 4 rocksdb: Options.num_levels: 7
2016-12-01 12:37:38.994334 7f0ac63f0800 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
2016-12-01 12:37:38.994354 7f0ac63f0800 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2016-12-01 12:37:38.994372 7f0ac63f0800 4 rocksdb: Options.compression_opts.window_bits: -14
2016-12-01 12:37:38.994389 7f0ac63f0800 4 rocksdb: Options.compression_opts.level: -1
2016-12-01 12:37:38.994406 7f0ac63f0800 4 rocksdb: Options.compression_opts.strategy: 0
2016-12-01 12:37:38.994423 7f0ac63f0800 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2016-12-01 12:37:38.994440 7f0ac63f0800 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2016-12-01 12:37:38.994457 7f0ac63f0800 4 rocksdb: Options.level0_stop_writes_trigger: 24
2016-12-01 12:37:38.994473 7f0ac63f0800 4 rocksdb: Options.target_file_size_base: 2097152
2016-12-01 12:37:38.994491 7f0ac63f0800 4 rocksdb: Options.target_file_size_multiplier: 1
2016-12-01 12:37:38.994509 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_base: 10485760
2016-12-01 12:37:38.994527 7f0ac63f0800 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2016-12-01 12:37:38.994544 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
2016-12-01 12:37:38.994561 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2016-12-01 12:37:38.994579 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2016-12-01 12:37:38.994596 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2016-12-01 12:37:38.994614 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2016-12-01 12:37:38.994639 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2016-12-01 12:37:38.994657 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2016-12-01 12:37:38.994675 7f0ac63f0800 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2016-12-01 12:37:38.994692 7f0ac63f0800 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2016-12-01 12:37:38.994709 7f0ac63f0800 4 rocksdb: Options.expanded_compaction_factor: 25
2016-12-01 12:37:38.994725 7f0ac63f0800 4 rocksdb: Options.source_compaction_factor: 1
2016-12-01 12:37:38.994742 7f0ac63f0800 4 rocksdb: Options.max_grandparent_overlap_factor: 10
2016-12-01 12:37:38.994759 7f0ac63f0800 4 rocksdb: Options.arena_block_size: 524288
2016-12-01 12:37:38.994775 7f0ac63f0800 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
2016-12-01 12:37:38.994792 7f0ac63f0800 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
2016-12-01 12:37:38.994810 7f0ac63f0800 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2016-12-01 12:37:38.994827 7f0ac63f0800 4 rocksdb: Options.disable_auto_compactions: 0
2016-12-01 12:37:38.994843 7f0ac63f0800 4 rocksdb: Options.filter_deletes: 0
2016-12-01 12:37:38.994865 7f0ac63f0800 4 rocksdb: Options.verify_checksums_in_compaction: 1
2016-12-01 12:37:38.994887 7f0ac63f0800 4 rocksdb: Options.compaction_style: 0
2016-12-01 12:37:38.994904 7f0ac63f0800 4 rocksdb: Options.compaction_pri: 0
2016-12-01 12:37:38.994922 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2016-12-01 12:37:38.994939 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2016-12-01 12:37:38.994956 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2016-12-01 12:37:38.994973 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2016-12-01 12:37:38.994991 7f0ac63f0800 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2016-12-01 12:37:38.995008 7f0ac63f0800 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2016-12-01 12:37:38.995025 7f0ac63f0800 4 rocksdb: Options.table_properties_collectors:
2016-12-01 12:37:38.995043 7f0ac63f0800 4 rocksdb: Options.inplace_update_support: 0
2016-12-01 12:37:38.995060 7f0ac63f0800 4 rocksdb: Options.inplace_update_num_locks: 10000
2016-12-01 12:37:38.995076 7f0ac63f0800 4 rocksdb: Options.min_partial_merge_operands: 2
2016-12-01 12:37:38.995093 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
2016-12-01 12:37:38.995111 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
2016-12-01 12:37:38.995130 7f0ac63f0800 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
2016-12-01 12:37:38.995148 7f0ac63f0800 4 rocksdb: Options.bloom_locality: 0
2016-12-01 12:37:38.995165 7f0ac63f0800 4 rocksdb: Options.max_successive_merges: 0
2016-12-01 12:37:38.995183 7f0ac63f0800 4 rocksdb: Options.optimize_fllters_for_hits: 0
2016-12-01 12:37:38.995200 7f0ac63f0800 4 rocksdb: Options.paranoid_file_checks: 0
2016-12-01 12:37:38.995217 7f0ac63f0800 4 rocksdb: Options.compaction_measure_io_stats: 0
2016-12-01 12:37:39.007146 7f0ac63f0800 4 rocksdb: Recovered from manifest file:/var/lib/ceph/osd/cephsml-24/db/MANIFEST-032684 succeeded,manifest_file_number is 32684, next_file_number is 32686, last_sequence is 29816718, log_number is 0,prev_log_number is 0,max_column_family is 0
2016-12-01 12:37:39.007161 7f0ac63f0800 4 rocksdb: Column family [default] (ID 0), log number is 32671
2016-12-01 12:37:39.012126 7f0ac63f0800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480585059012118, "job": 1, "event": "recovery_started", "log_files": [32673, 32676, 32679, 32682, 32685]}
2016-12-01 12:37:39.012152 7f0ac63f0800 4 rocksdb: Recovering log #32673 mode 0 skip-recovery 0
2016-12-01 12:37:39.012181 7f0ac63f0800 4 rocksdb: Recovering log #32676 mode 0 skip-recovery 0
2016-12-01 12:37:39.012198 7f0ac63f0800 4 rocksdb: Recovering log #32679 mode 0 skip-recovery 0
2016-12-01 12:37:39.012213 7f0ac63f0800 4 rocksdb: Recovering log #32682 mode 0 skip-recovery 0
2016-12-01 12:37:39.012228 7f0ac63f0800 4 rocksdb: Recovering log #32685 mode 0 skip-recovery 0
2016-12-01 12:37:39.012290 7f0ac63f0800 4 rocksdb: Creating manifest 32687
2016-12-01 12:37:39.044359 7f0ac63f0800 4 rocksdb: Deleting manifest 32684 current manifest 32687
2016-12-01 12:37:39.044699 7f0ac63f0800 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1480585059044692, "job": 1, "event": "recovery_finished"}
Updated by Yuri Gorshkov over 7 years ago
Please close this as duplicate, I forgot to add all the needed bug attributes before I hit submit :(
Updated by Sage Weil about 7 years ago
- Status changed from New to Won't Fix
this si expected; there was an ondisk format change right before kraken was released.
Actions