Project

General

Profile

Bug #16278 » ceph-osd-3.log

Mikaël Cluseau, 07/13/2016 08:47 AM

 
+ exec /usr/bin/docker run --name=ceph-osd-3 --rm -v /var/lib/ceph/osd/ceph-3:/var/lib/ceph/osd/ceph-3 -v /etc/ceph:/etc/ceph -v /dev:/dev --net=host --privileged --pid=host --entrypoint=/usr/bin/ceph-osd my-mirror/ceph:jewel -d -f --id 3
2016-07-13 08:41:42.267539 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-07-13 08:41:42.267674 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-07-13 08:41:42.267689 7f65cbfb18c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-osd, pid 25829
2016-07-13 08:41:42.268346 7f65cbfb18c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
2016-07-13 08:41:42.269768 7f65cbfb18c0 0 pidfile_write: ignore empty --pid-file
2016-07-13 08:41:42.271278 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-07-13 08:41:42.280370 7f65cbfb18c0 1 bluestore(/var/lib/ceph/osd/ceph-3) mount path /var/lib/ceph/osd/ceph-3
2016-07-13 08:41:42.280432 7f65cbfb18c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2016-07-13 08:41:42.281044 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2016-07-13 08:41:42.281162 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
2016-07-13 08:41:42.282194 7f65cbfb18c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2016-07-13 08:41:42.282731 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2016-07-13 08:41:42.282815 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
2016-07-13 08:41:42.282829 7f65cbfb18c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 931 GB
2016-07-13 08:41:42.282861 7f65cbfb18c0 1 bluefs mount
2016-07-13 08:41:42.418934 7f65cbfb18c0 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

2016-07-13 08:41:42.419001 7f65cbfb18c0 0 set rocksdb option compression = kNoCompression
2016-07-13 08:41:42.419009 7f65cbfb18c0 0 set rocksdb option max_write_buffer_number = 16
2016-07-13 08:41:42.419013 7f65cbfb18c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-07-13 08:41:42.419016 7f65cbfb18c0 0 set rocksdb option recycle_log_file_num = 16
2016-07-13 08:41:42.419031 7f65cbfb18c0 0 set rocksdb option compression = kNoCompression
2016-07-13 08:41:42.419043 7f65cbfb18c0 0 set rocksdb option max_write_buffer_number = 16
2016-07-13 08:41:42.419047 7f65cbfb18c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-07-13 08:41:42.419051 7f65cbfb18c0 0 set rocksdb option recycle_log_file_num = 16
2016-07-13 08:41:42.419131 7f65cbfb18c0 4 rocksdb: RocksDB version: 4.3.0

2016-07-13 08:41:42.419164 7f65cbfb18c0 4 rocksdb: Git sha rocksdb_build_git_sha:
2016-07-13 08:41:42.419178 7f65cbfb18c0 4 rocksdb: Compile date Jun 14 2016
2016-07-13 08:41:42.419194 7f65cbfb18c0 4 rocksdb: DB SUMMARY

2016-07-13 08:41:42.424230 7f65cbfb18c0 4 rocksdb: CURRENT file: CURRENT

2016-07-13 08:41:42.424237 7f65cbfb18c0 4 rocksdb: IDENTITY file: IDENTITY

2016-07-13 08:41:42.424240 7f65cbfb18c0 4 rocksdb: MANIFEST file: MANIFEST-028336 size: 9090 Bytes

2016-07-13 08:41:42.425117 7f65cbfb18c0 4 rocksdb: SST files in db dir, Total Num: 62, files: 000599.sst 000600.sst 000601.sst 000771.sst 000772.sst 000773.sst 000774.sst 000852.sst 000853.sst

2016-07-13 08:41:42.425124 7f65cbfb18c0 4 rocksdb: Write Ahead Log file in db: 028337.log size: 0 ;

2016-07-13 08:41:42.425339 7f65cbfb18c0 4 rocksdb: Options.error_if_exists: 0
2016-07-13 08:41:42.425346 7f65cbfb18c0 4 rocksdb: Options.create_if_missing: 0
2016-07-13 08:41:42.425348 7f65cbfb18c0 4 rocksdb: Options.paranoid_checks: 1
2016-07-13 08:41:42.425349 7f65cbfb18c0 4 rocksdb: Options.env: 0x563f9d7fcfe0
2016-07-13 08:41:42.425350 7f65cbfb18c0 4 rocksdb: Options.info_log: 0x563f9dd7c080
2016-07-13 08:41:42.425351 7f65cbfb18c0 4 rocksdb: Options.max_open_files: 5000
2016-07-13 08:41:42.425352 7f65cbfb18c0 4 rocksdb: Options.max_file_opening_threads: 1
2016-07-13 08:41:42.425353 7f65cbfb18c0 4 rocksdb: Options.max_total_wal_size: 0
2016-07-13 08:41:42.425353 7f65cbfb18c0 4 rocksdb: Options.disableDataSync: 0
2016-07-13 08:41:42.425354 7f65cbfb18c0 4 rocksdb: Options.use_fsync: 0
2016-07-13 08:41:42.425355 7f65cbfb18c0 4 rocksdb: Options.max_log_file_size: 0
2016-07-13 08:41:42.425356 7f65cbfb18c0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2016-07-13 08:41:42.425357 7f65cbfb18c0 4 rocksdb: Options.log_file_time_to_roll: 0
2016-07-13 08:41:42.425358 7f65cbfb18c0 4 rocksdb: Options.keep_log_file_num: 1000
2016-07-13 08:41:42.425359 7f65cbfb18c0 4 rocksdb: Options.recycle_log_file_num: 16
2016-07-13 08:41:42.425359 7f65cbfb18c0 4 rocksdb: Options.allow_os_buffer: 1
2016-07-13 08:41:42.425360 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_reads: 0
2016-07-13 08:41:42.425360 7f65cbfb18c0 4 rocksdb: Options.allow_fallocate: 1
2016-07-13 08:41:42.425361 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_writes: 0
2016-07-13 08:41:42.425362 7f65cbfb18c0 4 rocksdb: Options.create_missing_column_families: 0
2016-07-13 08:41:42.425362 7f65cbfb18c0 4 rocksdb: Options.db_log_dir:
2016-07-13 08:41:42.425363 7f65cbfb18c0 4 rocksdb: Options.wal_dir: db
2016-07-13 08:41:42.425364 7f65cbfb18c0 4 rocksdb: Options.table_cache_numshardbits: 4
2016-07-13 08:41:42.425364 7f65cbfb18c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2016-07-13 08:41:42.425365 7f65cbfb18c0 4 rocksdb: Options.max_background_compactions: 1
2016-07-13 08:41:42.425366 7f65cbfb18c0 4 rocksdb: Options.max_subcompactions: 1
2016-07-13 08:41:42.425367 7f65cbfb18c0 4 rocksdb: Options.max_background_flushes: 1
2016-07-13 08:41:42.425368 7f65cbfb18c0 4 rocksdb: Options.WAL_ttl_seconds: 0
2016-07-13 08:41:42.425368 7f65cbfb18c0 4 rocksdb: Options.WAL_size_limit_MB: 0
2016-07-13 08:41:42.425369 7f65cbfb18c0 4 rocksdb: Options.manifest_preallocation_size: 4194304
2016-07-13 08:41:42.425370 7f65cbfb18c0 4 rocksdb: Options.allow_os_buffer: 1
2016-07-13 08:41:42.425370 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_reads: 0
2016-07-13 08:41:42.425371 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_writes: 0
2016-07-13 08:41:42.425371 7f65cbfb18c0 4 rocksdb: Options.is_fd_close_on_exec: 1
2016-07-13 08:41:42.425372 7f65cbfb18c0 4 rocksdb: Options.stats_dump_period_sec: 600
2016-07-13 08:41:42.425373 7f65cbfb18c0 4 rocksdb: Options.advise_random_on_open: 1
2016-07-13 08:41:42.425373 7f65cbfb18c0 4 rocksdb: Options.db_write_buffer_size: 0d
2016-07-13 08:41:42.425374 7f65cbfb18c0 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
2016-07-13 08:41:42.425375 7f65cbfb18c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2016-07-13 08:41:42.425375 7f65cbfb18c0 4 rocksdb: Options.compaction_readahead_size: 0d
2016-07-13 08:41:42.425376 7f65cbfb18c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
2016-07-13 08:41:42.425377 7f65cbfb18c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
2016-07-13 08:41:42.425378 7f65cbfb18c0 4 rocksdb: Options.use_adaptive_mutex: 0
2016-07-13 08:41:42.425378 7f65cbfb18c0 4 rocksdb: Options.rate_limiter: (nil)
2016-07-13 08:41:42.425379 7f65cbfb18c0 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
2016-07-13 08:41:42.425381 7f65cbfb18c0 4 rocksdb: Options.bytes_per_sync: 0
2016-07-13 08:41:42.425381 7f65cbfb18c0 4 rocksdb: Options.wal_bytes_per_sync: 0
2016-07-13 08:41:42.425382 7f65cbfb18c0 4 rocksdb: Options.wal_recovery_mode: 0
2016-07-13 08:41:42.425382 7f65cbfb18c0 4 rocksdb: Options.enable_thread_tracking: 0
2016-07-13 08:41:42.425383 7f65cbfb18c0 4 rocksdb: Options.row_cache: None
2016-07-13 08:41:42.425384 7f65cbfb18c0 4 rocksdb: Options.wal_filter: None
2016-07-13 08:41:42.425385 7f65cbfb18c0 4 rocksdb: Compression algorithms supported:
2016-07-13 08:41:42.425386 7f65cbfb18c0 4 rocksdb: Snappy supported: 1
2016-07-13 08:41:42.425386 7f65cbfb18c0 4 rocksdb: Zlib supported: 1
2016-07-13 08:41:42.425387 7f65cbfb18c0 4 rocksdb: Bzip supported: 0
2016-07-13 08:41:42.425388 7f65cbfb18c0 4 rocksdb: LZ4 supported: 0
2016-07-13 08:41:42.425389 7f65cbfb18c0 4 rocksdb: Fast CRC32 supported: 0
2016-07-13 08:41:42.426065 7f65cbfb18c0 4 rocksdb: Recovering from manifest file: MANIFEST-028336

2016-07-13 08:41:42.426107 7f65cbfb18c0 4 rocksdb: --------------- Options for column family [default]:

2016-07-13 08:41:42.426118 7f65cbfb18c0 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2016-07-13 08:41:42.426119 7f65cbfb18c0 4 rocksdb: Options.merge_operator: None
2016-07-13 08:41:42.426120 7f65cbfb18c0 4 rocksdb: Options.compaction_filter: None
2016-07-13 08:41:42.426121 7f65cbfb18c0 4 rocksdb: Options.compaction_filter_factory: None
2016-07-13 08:41:42.426121 7f65cbfb18c0 4 rocksdb: Options.memtable_factory: SkipListFactory
2016-07-13 08:41:42.426122 7f65cbfb18c0 4 rocksdb: Options.table_factory: BlockBasedTable
2016-07-13 08:41:42.426135 7f65cbfb18c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563f9d5fc108)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x563f9d629840
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0

2016-07-13 08:41:42.426138 7f65cbfb18c0 4 rocksdb: Options.write_buffer_size: 4194304
2016-07-13 08:41:42.426139 7f65cbfb18c0 4 rocksdb: Options.max_write_buffer_number: 16
2016-07-13 08:41:42.426140 7f65cbfb18c0 4 rocksdb: Options.compression: NoCompression
2016-07-13 08:41:42.426141 7f65cbfb18c0 4 rocksdb: Options.prefix_extractor: nullptr
2016-07-13 08:41:42.426142 7f65cbfb18c0 4 rocksdb: Options.num_levels: 7
2016-07-13 08:41:42.426143 7f65cbfb18c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
2016-07-13 08:41:42.426143 7f65cbfb18c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2016-07-13 08:41:42.426144 7f65cbfb18c0 4 rocksdb: Options.compression_opts.window_bits: -14
2016-07-13 08:41:42.426145 7f65cbfb18c0 4 rocksdb: Options.compression_opts.level: -1
2016-07-13 08:41:42.426145 7f65cbfb18c0 4 rocksdb: Options.compression_opts.strategy: 0
2016-07-13 08:41:42.426146 7f65cbfb18c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2016-07-13 08:41:42.426147 7f65cbfb18c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2016-07-13 08:41:42.426147 7f65cbfb18c0 4 rocksdb: Options.level0_stop_writes_trigger: 24
2016-07-13 08:41:42.426148 7f65cbfb18c0 4 rocksdb: Options.target_file_size_base: 2097152
2016-07-13 08:41:42.426149 7f65cbfb18c0 4 rocksdb: Options.target_file_size_multiplier: 1
2016-07-13 08:41:42.426149 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_base: 10485760
2016-07-13 08:41:42.426150 7f65cbfb18c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2016-07-13 08:41:42.426151 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
2016-07-13 08:41:42.426151 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2016-07-13 08:41:42.426152 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2016-07-13 08:41:42.426153 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2016-07-13 08:41:42.426154 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2016-07-13 08:41:42.426154 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2016-07-13 08:41:42.426155 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2016-07-13 08:41:42.426156 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2016-07-13 08:41:42.426156 7f65cbfb18c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2016-07-13 08:41:42.426157 7f65cbfb18c0 4 rocksdb: Options.expanded_compaction_factor: 25
2016-07-13 08:41:42.426158 7f65cbfb18c0 4 rocksdb: Options.source_compaction_factor: 1
2016-07-13 08:41:42.426158 7f65cbfb18c0 4 rocksdb: Options.max_grandparent_overlap_factor: 10
2016-07-13 08:41:42.426159 7f65cbfb18c0 4 rocksdb: Options.arena_block_size: 524288
2016-07-13 08:41:42.426160 7f65cbfb18c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
2016-07-13 08:41:42.426160 7f65cbfb18c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
2016-07-13 08:41:42.426161 7f65cbfb18c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2016-07-13 08:41:42.426162 7f65cbfb18c0 4 rocksdb: Options.disable_auto_compactions: 0
2016-07-13 08:41:42.426162 7f65cbfb18c0 4 rocksdb: Options.filter_deletes: 0
2016-07-13 08:41:42.426163 7f65cbfb18c0 4 rocksdb: Options.verify_checksums_in_compaction: 1
2016-07-13 08:41:42.426164 7f65cbfb18c0 4 rocksdb: Options.compaction_style: 0
2016-07-13 08:41:42.426164 7f65cbfb18c0 4 rocksdb: Options.compaction_pri: 0
2016-07-13 08:41:42.426165 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2016-07-13 08:41:42.426165 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2016-07-13 08:41:42.426166 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2016-07-13 08:41:42.426167 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2016-07-13 08:41:42.426168 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2016-07-13 08:41:42.426169 7f65cbfb18c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2016-07-13 08:41:42.426170 7f65cbfb18c0 4 rocksdb: Options.table_properties_collectors:
2016-07-13 08:41:42.426171 7f65cbfb18c0 4 rocksdb: Options.inplace_update_support: 0
2016-07-13 08:41:42.426172 7f65cbfb18c0 4 rocksdb: Options.inplace_update_num_locks: 10000
2016-07-13 08:41:42.426172 7f65cbfb18c0 4 rocksdb: Options.min_partial_merge_operands: 2
2016-07-13 08:41:42.426173 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
2016-07-13 08:41:42.426174 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
2016-07-13 08:41:42.426174 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
2016-07-13 08:41:42.426175 7f65cbfb18c0 4 rocksdb: Options.bloom_locality: 0
2016-07-13 08:41:42.426176 7f65cbfb18c0 4 rocksdb: Options.max_successive_merges: 0
2016-07-13 08:41:42.426177 7f65cbfb18c0 4 rocksdb: Options.optimize_fllters_for_hits: 0
2016-07-13 08:41:42.426178 7f65cbfb18c0 4 rocksdb: Options.paranoid_file_checks: 0
2016-07-13 08:41:42.426179 7f65cbfb18c0 4 rocksdb: Options.compaction_measure_io_stats: 0
2016-07-13 08:41:42.433060 7f65cbfb18c0 4 rocksdb: Recovered from manifest file:db/MANIFEST-028336 succeeded,manifest_file_number is 28336, next_file_number is 28338, last_sequence is 3262218, log_number is 0,prev_log_number is 0,max_column_family is 0

2016-07-13 08:41:42.433070 7f65cbfb18c0 4 rocksdb: Column family [default] (ID 0), log number is 28335

2016-07-13 08:41:42.434443 7f65cbfb18c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1468399302434436, "job": 1, "event": "recovery_started", "log_files": [28337]}
2016-07-13 08:41:42.434455 7f65cbfb18c0 4 rocksdb: Recovering log #28337 mode 0 skip-recovery 0
2016-07-13 08:41:42.434465 7f65cbfb18c0 4 rocksdb: Creating manifest 28339

2016-07-13 08:41:42.436671 7f65cbfb18c0 4 rocksdb: Deleting manifest 28336 current manifest 28339

2016-07-13 08:41:42.436696 7f65cbfb18c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1468399302436692, "job": 1, "event": "recovery_finished"}
2016-07-13 08:41:42.449414 7f65cbfb18c0 4 rocksdb: DB pointer 0x563f9dd8e000
2016-07-13 08:41:42.449435 7f65cbfb18c0 1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=16
2016-07-13 08:41:42.449441 7f65cbfb18c0 1 freelist init prefix B
2016-07-13 08:41:42.462119 7f65cbfb18c0 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
2016-07-13 08:41:42.466770 7f65cbfb18c0 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65cbfb18c0 time 2016-07-13 08:41:42.468056
osd/OSD.h: 885: FAILED assert(ret)
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x563f92a23040]
2: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
3: (OSD::init()+0x1f91) [0x563f9234d8b1]
4: (main()+0x2ea5) [0x563f922bee55]
5: (__libc_start_main()+0xf0) [0x7f65c8df6830]
6: (_start()+0x29) [0x563f923004e9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-07-13 08:41:42.469640 7f65cbfb18c0 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65cbfb18c0 time 2016-07-13 08:41:42.468056
osd/OSD.h: 885: FAILED assert(ret)

ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x563f92a23040]
2: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
3: (OSD::init()+0x1f91) [0x563f9234d8b1]
4: (main()+0x2ea5) [0x563f922bee55]
5: (__libc_start_main()+0xf0) [0x7f65c8df6830]
6: (_start()+0x29) [0x563f923004e9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-215> 2016-07-13 08:41:42.265568 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command perfcounters_dump hook 0x563f9d600030
-214> 2016-07-13 08:41:42.265594 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command 1 hook 0x563f9d600030
-213> 2016-07-13 08:41:42.265600 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command perf dump hook 0x563f9d600030
-212> 2016-07-13 08:41:42.265605 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command perfcounters_schema hook 0x563f9d600030
-211> 2016-07-13 08:41:42.265609 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command 2 hook 0x563f9d600030
-210> 2016-07-13 08:41:42.265612 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command perf schema hook 0x563f9d600030
-209> 2016-07-13 08:41:42.265616 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command perf reset hook 0x563f9d600030
-208> 2016-07-13 08:41:42.265618 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command config show hook 0x563f9d600030
-207> 2016-07-13 08:41:42.265622 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command config set hook 0x563f9d600030
-206> 2016-07-13 08:41:42.265627 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command config get hook 0x563f9d600030
-205> 2016-07-13 08:41:42.265632 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command config diff hook 0x563f9d600030
-204> 2016-07-13 08:41:42.265636 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command log flush hook 0x563f9d600030
-203> 2016-07-13 08:41:42.265638 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command log dump hook 0x563f9d600030
-202> 2016-07-13 08:41:42.265643 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command log reopen hook 0x563f9d600030
-201> 2016-07-13 08:41:42.267539 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-200> 2016-07-13 08:41:42.267674 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-199> 2016-07-13 08:41:42.267689 7f65cbfb18c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-osd, pid 25829
-198> 2016-07-13 08:41:42.268329 7f65cbfb18c0 5 object store type is bluestore
-197> 2016-07-13 08:41:42.268346 7f65cbfb18c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

-196> 2016-07-13 08:41:42.269699 7f65cbfb18c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6808/25829 need_addr=1
-195> 2016-07-13 08:41:42.269728 7f65cbfb18c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/25829 need_addr=1
-194> 2016-07-13 08:41:42.269747 7f65cbfb18c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6810/25829 need_addr=1
-193> 2016-07-13 08:41:42.269765 7f65cbfb18c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6811/25829 need_addr=1
-192> 2016-07-13 08:41:42.269768 7f65cbfb18c0 0 pidfile_write: ignore empty --pid-file
-191> 2016-07-13 08:41:42.271278 7f65cbfb18c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-190> 2016-07-13 08:41:42.271281 7f65cbfb18c0 5 asok(0x563f9d6c0000) init /var/run/ceph/ceph-osd.3.asok
-189> 2016-07-13 08:41:42.271289 7f65cbfb18c0 5 asok(0x563f9d6c0000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
-188> 2016-07-13 08:41:42.271641 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command 0 hook 0x563f9d5fc0a8
-187> 2016-07-13 08:41:42.271646 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command version hook 0x563f9d5fc0a8
-186> 2016-07-13 08:41:42.271649 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command git_version hook 0x563f9d5fc0a8
-185> 2016-07-13 08:41:42.271652 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command help hook 0x563f9d600110
-184> 2016-07-13 08:41:42.271655 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command get_command_descriptions hook 0x563f9d600120
-183> 2016-07-13 08:41:42.271715 7f65cbfb18c0 10 monclient(hunting): build_initial_monmap
-182> 2016-07-13 08:41:42.271718 7f65c6391700 5 asok(0x563f9d6c0000) entry start
-181> 2016-07-13 08:41:42.279894 7f65cbfb18c0 5 adding auth protocol: cephx
-180> 2016-07-13 08:41:42.279903 7f65cbfb18c0 5 adding auth protocol: cephx
-179> 2016-07-13 08:41:42.280062 7f65cbfb18c0 5 asok(0x563f9d6c0000) register_command objecter_requests hook 0x563f9d6001d0
-178> 2016-07-13 08:41:42.280141 7f65cbfb18c0 1 -- 0.0.0.0:6808/25829 messenger.start
-177> 2016-07-13 08:41:42.280179 7f65cbfb18c0 1 -- :/0 messenger.start
-176> 2016-07-13 08:41:42.280205 7f65cbfb18c0 1 -- 0.0.0.0:6811/25829 messenger.start
-175> 2016-07-13 08:41:42.280231 7f65cbfb18c0 1 -- 0.0.0.0:6810/25829 messenger.start
-174> 2016-07-13 08:41:42.280255 7f65cbfb18c0 1 -- 0.0.0.0:6809/25829 messenger.start
-173> 2016-07-13 08:41:42.280277 7f65cbfb18c0 1 -- :/0 messenger.start
-172> 2016-07-13 08:41:42.280359 7f65cbfb18c0 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
-171> 2016-07-13 08:41:42.280370 7f65cbfb18c0 1 bluestore(/var/lib/ceph/osd/ceph-3) mount path /var/lib/ceph/osd/ceph-3
-170> 2016-07-13 08:41:42.280432 7f65cbfb18c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
-169> 2016-07-13 08:41:42.281044 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
-168> 2016-07-13 08:41:42.281162 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
-167> 2016-07-13 08:41:42.282194 7f65cbfb18c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
-166> 2016-07-13 08:41:42.282731 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
-165> 2016-07-13 08:41:42.282815 7f65cbfb18c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
-164> 2016-07-13 08:41:42.282829 7f65cbfb18c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 931 GB
-163> 2016-07-13 08:41:42.282861 7f65cbfb18c0 1 bluefs mount
-162> 2016-07-13 08:41:42.418934 7f65cbfb18c0 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

-161> 2016-07-13 08:41:42.419001 7f65cbfb18c0 0 set rocksdb option compression = kNoCompression
-160> 2016-07-13 08:41:42.419009 7f65cbfb18c0 0 set rocksdb option max_write_buffer_number = 16
-159> 2016-07-13 08:41:42.419013 7f65cbfb18c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
-158> 2016-07-13 08:41:42.419016 7f65cbfb18c0 0 set rocksdb option recycle_log_file_num = 16
-157> 2016-07-13 08:41:42.419031 7f65cbfb18c0 0 set rocksdb option compression = kNoCompression
-156> 2016-07-13 08:41:42.419043 7f65cbfb18c0 0 set rocksdb option max_write_buffer_number = 16
-155> 2016-07-13 08:41:42.419047 7f65cbfb18c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
-154> 2016-07-13 08:41:42.419051 7f65cbfb18c0 0 set rocksdb option recycle_log_file_num = 16
-153> 2016-07-13 08:41:42.419131 7f65cbfb18c0 4 rocksdb: RocksDB version: 4.3.0

-152> 2016-07-13 08:41:42.419164 7f65cbfb18c0 4 rocksdb: Git sha rocksdb_build_git_sha:
-151> 2016-07-13 08:41:42.419178 7f65cbfb18c0 4 rocksdb: Compile date Jun 14 2016
-150> 2016-07-13 08:41:42.419194 7f65cbfb18c0 4 rocksdb: DB SUMMARY

-149> 2016-07-13 08:41:42.424230 7f65cbfb18c0 4 rocksdb: CURRENT file: CURRENT

-148> 2016-07-13 08:41:42.424237 7f65cbfb18c0 4 rocksdb: IDENTITY file: IDENTITY

-147> 2016-07-13 08:41:42.424240 7f65cbfb18c0 4 rocksdb: MANIFEST file: MANIFEST-028336 size: 9090 Bytes

-146> 2016-07-13 08:41:42.425117 7f65cbfb18c0 4 rocksdb: SST files in db dir, Total Num: 62, files: 000599.sst 000600.sst 000601.sst 000771.sst 000772.sst 000773.sst 000774.sst 000852.sst 000853.sst

-145> 2016-07-13 08:41:42.425124 7f65cbfb18c0 4 rocksdb: Write Ahead Log file in db: 028337.log size: 0 ;

-144> 2016-07-13 08:41:42.425339 7f65cbfb18c0 4 rocksdb: Options.error_if_exists: 0
-143> 2016-07-13 08:41:42.425346 7f65cbfb18c0 4 rocksdb: Options.create_if_missing: 0
-142> 2016-07-13 08:41:42.425348 7f65cbfb18c0 4 rocksdb: Options.paranoid_checks: 1
-141> 2016-07-13 08:41:42.425349 7f65cbfb18c0 4 rocksdb: Options.env: 0x563f9d7fcfe0
-140> 2016-07-13 08:41:42.425350 7f65cbfb18c0 4 rocksdb: Options.info_log: 0x563f9dd7c080
-139> 2016-07-13 08:41:42.425351 7f65cbfb18c0 4 rocksdb: Options.max_open_files: 5000
-138> 2016-07-13 08:41:42.425352 7f65cbfb18c0 4 rocksdb: Options.max_file_opening_threads: 1
-137> 2016-07-13 08:41:42.425353 7f65cbfb18c0 4 rocksdb: Options.max_total_wal_size: 0
-136> 2016-07-13 08:41:42.425353 7f65cbfb18c0 4 rocksdb: Options.disableDataSync: 0
-135> 2016-07-13 08:41:42.425354 7f65cbfb18c0 4 rocksdb: Options.use_fsync: 0
-134> 2016-07-13 08:41:42.425355 7f65cbfb18c0 4 rocksdb: Options.max_log_file_size: 0
-133> 2016-07-13 08:41:42.425356 7f65cbfb18c0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-132> 2016-07-13 08:41:42.425357 7f65cbfb18c0 4 rocksdb: Options.log_file_time_to_roll: 0
-131> 2016-07-13 08:41:42.425358 7f65cbfb18c0 4 rocksdb: Options.keep_log_file_num: 1000
-130> 2016-07-13 08:41:42.425359 7f65cbfb18c0 4 rocksdb: Options.recycle_log_file_num: 16
-129> 2016-07-13 08:41:42.425359 7f65cbfb18c0 4 rocksdb: Options.allow_os_buffer: 1
-128> 2016-07-13 08:41:42.425360 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_reads: 0
-127> 2016-07-13 08:41:42.425360 7f65cbfb18c0 4 rocksdb: Options.allow_fallocate: 1
-126> 2016-07-13 08:41:42.425361 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_writes: 0
-125> 2016-07-13 08:41:42.425362 7f65cbfb18c0 4 rocksdb: Options.create_missing_column_families: 0
-124> 2016-07-13 08:41:42.425362 7f65cbfb18c0 4 rocksdb: Options.db_log_dir:
-123> 2016-07-13 08:41:42.425363 7f65cbfb18c0 4 rocksdb: Options.wal_dir: db
-122> 2016-07-13 08:41:42.425364 7f65cbfb18c0 4 rocksdb: Options.table_cache_numshardbits: 4
-121> 2016-07-13 08:41:42.425364 7f65cbfb18c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-120> 2016-07-13 08:41:42.425365 7f65cbfb18c0 4 rocksdb: Options.max_background_compactions: 1
-119> 2016-07-13 08:41:42.425366 7f65cbfb18c0 4 rocksdb: Options.max_subcompactions: 1
-118> 2016-07-13 08:41:42.425367 7f65cbfb18c0 4 rocksdb: Options.max_background_flushes: 1
-117> 2016-07-13 08:41:42.425368 7f65cbfb18c0 4 rocksdb: Options.WAL_ttl_seconds: 0
-116> 2016-07-13 08:41:42.425368 7f65cbfb18c0 4 rocksdb: Options.WAL_size_limit_MB: 0
-115> 2016-07-13 08:41:42.425369 7f65cbfb18c0 4 rocksdb: Options.manifest_preallocation_size: 4194304
-114> 2016-07-13 08:41:42.425370 7f65cbfb18c0 4 rocksdb: Options.allow_os_buffer: 1
-113> 2016-07-13 08:41:42.425370 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_reads: 0
-112> 2016-07-13 08:41:42.425371 7f65cbfb18c0 4 rocksdb: Options.allow_mmap_writes: 0
-111> 2016-07-13 08:41:42.425371 7f65cbfb18c0 4 rocksdb: Options.is_fd_close_on_exec: 1
-110> 2016-07-13 08:41:42.425372 7f65cbfb18c0 4 rocksdb: Options.stats_dump_period_sec: 600
-109> 2016-07-13 08:41:42.425373 7f65cbfb18c0 4 rocksdb: Options.advise_random_on_open: 1
-108> 2016-07-13 08:41:42.425373 7f65cbfb18c0 4 rocksdb: Options.db_write_buffer_size: 0d
-107> 2016-07-13 08:41:42.425374 7f65cbfb18c0 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
-106> 2016-07-13 08:41:42.425375 7f65cbfb18c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-105> 2016-07-13 08:41:42.425375 7f65cbfb18c0 4 rocksdb: Options.compaction_readahead_size: 0d
-104> 2016-07-13 08:41:42.425376 7f65cbfb18c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
-103> 2016-07-13 08:41:42.425377 7f65cbfb18c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
-102> 2016-07-13 08:41:42.425378 7f65cbfb18c0 4 rocksdb: Options.use_adaptive_mutex: 0
-101> 2016-07-13 08:41:42.425378 7f65cbfb18c0 4 rocksdb: Options.rate_limiter: (nil)
-100> 2016-07-13 08:41:42.425379 7f65cbfb18c0 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
-99> 2016-07-13 08:41:42.425381 7f65cbfb18c0 4 rocksdb: Options.bytes_per_sync: 0
-98> 2016-07-13 08:41:42.425381 7f65cbfb18c0 4 rocksdb: Options.wal_bytes_per_sync: 0
-97> 2016-07-13 08:41:42.425382 7f65cbfb18c0 4 rocksdb: Options.wal_recovery_mode: 0
-96> 2016-07-13 08:41:42.425382 7f65cbfb18c0 4 rocksdb: Options.enable_thread_tracking: 0
-95> 2016-07-13 08:41:42.425383 7f65cbfb18c0 4 rocksdb: Options.row_cache: None
-94> 2016-07-13 08:41:42.425384 7f65cbfb18c0 4 rocksdb: Options.wal_filter: None
-93> 2016-07-13 08:41:42.425385 7f65cbfb18c0 4 rocksdb: Compression algorithms supported:
-92> 2016-07-13 08:41:42.425386 7f65cbfb18c0 4 rocksdb: Snappy supported: 1
-91> 2016-07-13 08:41:42.425386 7f65cbfb18c0 4 rocksdb: Zlib supported: 1
-90> 2016-07-13 08:41:42.425387 7f65cbfb18c0 4 rocksdb: Bzip supported: 0
-89> 2016-07-13 08:41:42.425388 7f65cbfb18c0 4 rocksdb: LZ4 supported: 0
-88> 2016-07-13 08:41:42.425389 7f65cbfb18c0 4 rocksdb: Fast CRC32 supported: 0
-87> 2016-07-13 08:41:42.426065 7f65cbfb18c0 4 rocksdb: Recovering from manifest file: MANIFEST-028336

-86> 2016-07-13 08:41:42.426107 7f65cbfb18c0 4 rocksdb: --------------- Options for column family [default]:

-85> 2016-07-13 08:41:42.426118 7f65cbfb18c0 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
-84> 2016-07-13 08:41:42.426119 7f65cbfb18c0 4 rocksdb: Options.merge_operator: None
-83> 2016-07-13 08:41:42.426120 7f65cbfb18c0 4 rocksdb: Options.compaction_filter: None
-82> 2016-07-13 08:41:42.426121 7f65cbfb18c0 4 rocksdb: Options.compaction_filter_factory: None
-81> 2016-07-13 08:41:42.426121 7f65cbfb18c0 4 rocksdb: Options.memtable_factory: SkipListFactory
-80> 2016-07-13 08:41:42.426122 7f65cbfb18c0 4 rocksdb: Options.table_factory: BlockBasedTable
-79> 2016-07-13 08:41:42.426135 7f65cbfb18c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563f9d5fc108)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x563f9d629840
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0

-78> 2016-07-13 08:41:42.426138 7f65cbfb18c0 4 rocksdb: Options.write_buffer_size: 4194304
-77> 2016-07-13 08:41:42.426139 7f65cbfb18c0 4 rocksdb: Options.max_write_buffer_number: 16
-76> 2016-07-13 08:41:42.426140 7f65cbfb18c0 4 rocksdb: Options.compression: NoCompression
-75> 2016-07-13 08:41:42.426141 7f65cbfb18c0 4 rocksdb: Options.prefix_extractor: nullptr
-74> 2016-07-13 08:41:42.426142 7f65cbfb18c0 4 rocksdb: Options.num_levels: 7
-73> 2016-07-13 08:41:42.426143 7f65cbfb18c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
-72> 2016-07-13 08:41:42.426143 7f65cbfb18c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-71> 2016-07-13 08:41:42.426144 7f65cbfb18c0 4 rocksdb: Options.compression_opts.window_bits: -14
-70> 2016-07-13 08:41:42.426145 7f65cbfb18c0 4 rocksdb: Options.compression_opts.level: -1
-69> 2016-07-13 08:41:42.426145 7f65cbfb18c0 4 rocksdb: Options.compression_opts.strategy: 0
-68> 2016-07-13 08:41:42.426146 7f65cbfb18c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-67> 2016-07-13 08:41:42.426147 7f65cbfb18c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-66> 2016-07-13 08:41:42.426147 7f65cbfb18c0 4 rocksdb: Options.level0_stop_writes_trigger: 24
-65> 2016-07-13 08:41:42.426148 7f65cbfb18c0 4 rocksdb: Options.target_file_size_base: 2097152
-64> 2016-07-13 08:41:42.426149 7f65cbfb18c0 4 rocksdb: Options.target_file_size_multiplier: 1
-63> 2016-07-13 08:41:42.426149 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_base: 10485760
-62> 2016-07-13 08:41:42.426150 7f65cbfb18c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-61> 2016-07-13 08:41:42.426151 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
-60> 2016-07-13 08:41:42.426151 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-59> 2016-07-13 08:41:42.426152 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-58> 2016-07-13 08:41:42.426153 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-57> 2016-07-13 08:41:42.426154 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-56> 2016-07-13 08:41:42.426154 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-55> 2016-07-13 08:41:42.426155 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-54> 2016-07-13 08:41:42.426156 7f65cbfb18c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-53> 2016-07-13 08:41:42.426156 7f65cbfb18c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-52> 2016-07-13 08:41:42.426157 7f65cbfb18c0 4 rocksdb: Options.expanded_compaction_factor: 25
-51> 2016-07-13 08:41:42.426158 7f65cbfb18c0 4 rocksdb: Options.source_compaction_factor: 1
-50> 2016-07-13 08:41:42.426158 7f65cbfb18c0 4 rocksdb: Options.max_grandparent_overlap_factor: 10
-49> 2016-07-13 08:41:42.426159 7f65cbfb18c0 4 rocksdb: Options.arena_block_size: 524288
-48> 2016-07-13 08:41:42.426160 7f65cbfb18c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
-47> 2016-07-13 08:41:42.426160 7f65cbfb18c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
-46> 2016-07-13 08:41:42.426161 7f65cbfb18c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
-45> 2016-07-13 08:41:42.426162 7f65cbfb18c0 4 rocksdb: Options.disable_auto_compactions: 0
-44> 2016-07-13 08:41:42.426162 7f65cbfb18c0 4 rocksdb: Options.filter_deletes: 0
-43> 2016-07-13 08:41:42.426163 7f65cbfb18c0 4 rocksdb: Options.verify_checksums_in_compaction: 1
-42> 2016-07-13 08:41:42.426164 7f65cbfb18c0 4 rocksdb: Options.compaction_style: 0
-41> 2016-07-13 08:41:42.426164 7f65cbfb18c0 4 rocksdb: Options.compaction_pri: 0
-40> 2016-07-13 08:41:42.426165 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-39> 2016-07-13 08:41:42.426165 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-38> 2016-07-13 08:41:42.426166 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-37> 2016-07-13 08:41:42.426167 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-36> 2016-07-13 08:41:42.426168 7f65cbfb18c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-35> 2016-07-13 08:41:42.426169 7f65cbfb18c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-34> 2016-07-13 08:41:42.426170 7f65cbfb18c0 4 rocksdb: Options.table_properties_collectors:
-33> 2016-07-13 08:41:42.426171 7f65cbfb18c0 4 rocksdb: Options.inplace_update_support: 0
-32> 2016-07-13 08:41:42.426172 7f65cbfb18c0 4 rocksdb: Options.inplace_update_num_locks: 10000
-31> 2016-07-13 08:41:42.426172 7f65cbfb18c0 4 rocksdb: Options.min_partial_merge_operands: 2
-30> 2016-07-13 08:41:42.426173 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
-29> 2016-07-13 08:41:42.426174 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
-28> 2016-07-13 08:41:42.426174 7f65cbfb18c0 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
-27> 2016-07-13 08:41:42.426175 7f65cbfb18c0 4 rocksdb: Options.bloom_locality: 0
-26> 2016-07-13 08:41:42.426176 7f65cbfb18c0 4 rocksdb: Options.max_successive_merges: 0
-25> 2016-07-13 08:41:42.426177 7f65cbfb18c0 4 rocksdb: Options.optimize_fllters_for_hits: 0
-24> 2016-07-13 08:41:42.426178 7f65cbfb18c0 4 rocksdb: Options.paranoid_file_checks: 0
-23> 2016-07-13 08:41:42.426179 7f65cbfb18c0 4 rocksdb: Options.compaction_measure_io_stats: 0
-22> 2016-07-13 08:41:42.433060 7f65cbfb18c0 4 rocksdb: Recovered from manifest file:db/MANIFEST-028336 succeeded,manifest_file_number is 28336, next_file_number is 28338, last_sequence is 3262218, log_number is 0,prev_log_number is 0,max_column_family is 0

-21> 2016-07-13 08:41:42.433070 7f65cbfb18c0 4 rocksdb: Column family [default] (ID 0), log number is 28335

-20> 2016-07-13 08:41:42.434443 7f65cbfb18c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1468399302434436, "job": 1, "event": "recovery_started", "log_files": [28337]}
-19> 2016-07-13 08:41:42.434455 7f65cbfb18c0 4 rocksdb: Recovering log #28337 mode 0 skip-recovery 0
-18> 2016-07-13 08:41:42.434465 7f65cbfb18c0 4 rocksdb: Creating manifest 28339

-17> 2016-07-13 08:41:42.436671 7f65cbfb18c0 4 rocksdb: Deleting manifest 28336 current manifest 28339

-16> 2016-07-13 08:41:42.436696 7f65cbfb18c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1468399302436692, "job": 1, "event": "recovery_finished"}
-15> 2016-07-13 08:41:42.446643 7f65cbfb18c0 5 rocksdb: [JOB 2] Delete db//028337.log type=0 #28337 -- OK

-14> 2016-07-13 08:41:42.449414 7f65cbfb18c0 4 rocksdb: DB pointer 0x563f9dd8e000
-13> 2016-07-13 08:41:42.449435 7f65cbfb18c0 1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=16
-12> 2016-07-13 08:41:42.449441 7f65cbfb18c0 1 freelist init prefix B
-11> 2016-07-13 08:41:42.459672 7f65cbfb18c0 2 osd.3 0 boot
-10> 2016-07-13 08:41:42.461946 7f65cbfb18c0 1 <cls> cls/version/cls_version.cc:228: Loaded version class!
-9> 2016-07-13 08:41:42.462119 7f65cbfb18c0 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
-8> 2016-07-13 08:41:42.464367 7f65cbfb18c0 1 <cls> cls/refcount/cls_refcount.cc:232: Loaded refcount class!
-7> 2016-07-13 08:41:42.464487 7f65cbfb18c0 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-6> 2016-07-13 08:41:42.464601 7f65cbfb18c0 1 <cls> cls/log/cls_log.cc:317: Loaded log class!
-5> 2016-07-13 08:41:42.464915 7f65cbfb18c0 1 <cls> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class!
-4> 2016-07-13 08:41:42.466521 7f65cbfb18c0 1 <cls> cls/rgw/cls_rgw.cc:3207: Loaded rgw class!
-3> 2016-07-13 08:41:42.466659 7f65cbfb18c0 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-2> 2016-07-13 08:41:42.466770 7f65cbfb18c0 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
-1> 2016-07-13 08:41:42.466996 7f65cbfb18c0 1 <cls> cls/user/cls_user.cc:375: Loaded user class!
0> 2016-07-13 08:41:42.469640 7f65cbfb18c0 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65cbfb18c0 time 2016-07-13 08:41:42.468056
osd/OSD.h: 885: FAILED assert(ret)

ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x563f92a23040]
2: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
3: (OSD::init()+0x1f91) [0x563f9234d8b1]
4: (main()+0x2ea5) [0x563f922bee55]
5: (__libc_start_main()+0xf0) [0x7f65c8df6830]
6: (_start()+0x29) [0x563f923004e9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
*** Caught signal (Aborted) **
in thread 7f65cbfb18c0 thread_name:ceph-osd
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x96bdde) [0x563f9291edde]
2: (()+0x113d0) [0x7f65cae6b3d0]
3: (gsignal()+0x38) [0x7f65c8e0b418]
4: (abort()+0x16a) [0x7f65c8e0d01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x563f92a2322b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
7: (OSD::init()+0x1f91) [0x563f9234d8b1]
8: (main()+0x2ea5) [0x563f922bee55]
9: (__libc_start_main()+0xf0) [0x7f65c8df6830]
10: (_start()+0x29) [0x563f923004e9]
2016-07-13 08:41:42.472325 7f65cbfb18c0 -1 *** Caught signal (Aborted) **
in thread 7f65cbfb18c0 thread_name:ceph-osd

ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x96bdde) [0x563f9291edde]
2: (()+0x113d0) [0x7f65cae6b3d0]
3: (gsignal()+0x38) [0x7f65c8e0b418]
4: (abort()+0x16a) [0x7f65c8e0d01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x563f92a2322b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
7: (OSD::init()+0x1f91) [0x563f9234d8b1]
8: (main()+0x2ea5) [0x563f922bee55]
9: (__libc_start_main()+0xf0) [0x7f65c8df6830]
10: (_start()+0x29) [0x563f923004e9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2016-07-13 08:41:42.472325 7f65cbfb18c0 -1 *** Caught signal (Aborted) **
in thread 7f65cbfb18c0 thread_name:ceph-osd

ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x96bdde) [0x563f9291edde]
2: (()+0x113d0) [0x7f65cae6b3d0]
3: (gsignal()+0x38) [0x7f65c8e0b418]
4: (abort()+0x16a) [0x7f65c8e0d01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x563f92a2322b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x563f9239d93d]
7: (OSD::init()+0x1f91) [0x563f9234d8b1]
8: (main()+0x2ea5) [0x563f922bee55]
9: (__libc_start_main()+0xf0) [0x7f65c8df6830]
10: (_start()+0x29) [0x563f923004e9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
(3-3/3)