2017-09-26 14:21:27.323230 7f53bdff9e80 0 ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable), process (unknown), pid 1712100 2017-09-26 14:21:27.325259 7f53bdff9e80 -1 Public network was set, but cluster network was not set 2017-09-26 14:21:27.325274 7f53bdff9e80 -1 Using public network also for cluster network 2017-09-26 14:21:27.337739 7f53bdff9e80 0 load: jerasure load: lrc load: isa 2017-09-26 14:21:27.337825 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel 2017-09-26 14:21:27.337837 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block 2017-09-26 14:21:27.338169 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational 2017-09-26 14:21:27.364552 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _set_cache_sizes cache_size 268435456 meta 0.01 kv 0.99 data 0 2017-09-26 14:21:27.364649 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) close 2017-09-26 14:21:27.610678 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _mount path /var/lib/ceph/osd/ceph-113 2017-09-26 14:21:27.610750 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel 2017-09-26 14:21:27.610759 7f53bdff9e80 1 bdev(0x555968656fc0 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block 2017-09-26 14:21:27.610980 7f53bdff9e80 1 bdev(0x555968656fc0 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational 2017-09-26 14:21:27.614527 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _set_cache_sizes cache_size 268435456 meta 0.01 kv 0.99 data 0 2017-09-26 14:21:27.614610 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel 2017-09-26 14:21:27.614618 7f53bdff9e80 1 bdev(0x555968657200 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block 2017-09-26 14:21:27.614829 7f53bdff9e80 1 bdev(0x555968657200 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational 2017-09-26 14:21:27.614838 7f53bdff9e80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-113/block size 1862 GB 2017-09-26 14:21:27.614884 7f53bdff9e80 1 bluefs mount 2017-09-26 14:21:27.737451 7f53bdff9e80 0 set rocksdb option compaction_readahead_size = 2097152 2017-09-26 14:21:27.737483 7f53bdff9e80 0 set rocksdb option compression = kNoCompression 2017-09-26 14:21:27.737490 7f53bdff9e80 0 set rocksdb option max_write_buffer_number = 4 2017-09-26 14:21:27.737495 7f53bdff9e80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-09-26 14:21:27.737498 7f53bdff9e80 0 set rocksdb option recycle_log_file_num = 4 2017-09-26 14:21:27.737502 7f53bdff9e80 0 set rocksdb option writable_file_max_buffer_size = 0 2017-09-26 14:21:27.737507 7f53bdff9e80 0 set rocksdb option write_buffer_size = 268435456 2017-09-26 14:21:27.737528 7f53bdff9e80 0 set rocksdb option compaction_readahead_size = 2097152 2017-09-26 14:21:27.737533 7f53bdff9e80 0 set rocksdb option compression = kNoCompression 2017-09-26 14:21:27.737537 7f53bdff9e80 0 set rocksdb option max_write_buffer_number = 4 2017-09-26 14:21:27.737540 7f53bdff9e80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-09-26 14:21:27.737544 7f53bdff9e80 0 set rocksdb option recycle_log_file_num = 4 2017-09-26 14:21:27.737547 7f53bdff9e80 0 set rocksdb option writable_file_max_buffer_size = 0 2017-09-26 14:21:27.737551 7f53bdff9e80 0 set rocksdb option write_buffer_size = 268435456 2017-09-26 14:21:27.737689 7f53bdff9e80 4 rocksdb: RocksDB version: 5.4.0 2017-09-26 14:21:27.737702 7f53bdff9e80 4 rocksdb: Git sha rocksdb_build_git_sha:@bac0af364a2c742ea9bb63199ddeee0edacdb2ac@ 2017-09-26 14:21:27.737705 7f53bdff9e80 4 rocksdb: Compile date Sep 23 2017 2017-09-26 14:21:27.737708 7f53bdff9e80 4 rocksdb: DB SUMMARY 2017-09-26 14:21:27.737737 7f53bdff9e80 4 rocksdb: CURRENT file: CURRENT 2017-09-26 14:21:27.737742 7f53bdff9e80 4 rocksdb: IDENTITY file: IDENTITY 2017-09-26 14:21:27.737745 7f53bdff9e80 4 rocksdb: MANIFEST file: MANIFEST-000260 size: 15810 Bytes 2017-09-26 14:21:27.737748 7f53bdff9e80 4 rocksdb: SST files in db dir, Total Num: 18, files: 000313.sst 000314.sst 000317.sst 000354.sst 000364.sst 000366.sst 000369.sst 000370.sst 000371.sst 2017-09-26 14:21:27.737750 7f53bdff9e80 4 rocksdb: Write Ahead Log file in db: 000361.log size: 242511746 ; 000381.log size: 243278096 ; 2017-09-26 14:21:27.737753 7f53bdff9e80 4 rocksdb: Options.error_if_exists: 0 2017-09-26 14:21:27.737754 7f53bdff9e80 4 rocksdb: Options.create_if_missing: 0 2017-09-26 14:21:27.737756 7f53bdff9e80 4 rocksdb: Options.paranoid_checks: 1 2017-09-26 14:21:27.737758 7f53bdff9e80 4 rocksdb: Options.env: 0x555968a09f40 2017-09-26 14:21:27.737759 7f53bdff9e80 4 rocksdb: Options.info_log: 0x5559687165a0 2017-09-26 14:21:27.737762 7f53bdff9e80 4 rocksdb: Options.max_open_files: -1 2017-09-26 14:21:27.737764 7f53bdff9e80 4 rocksdb: Options.max_file_opening_threads: 16 2017-09-26 14:21:27.737765 7f53bdff9e80 4 rocksdb: Options.use_fsync: 0 2017-09-26 14:21:27.737766 7f53bdff9e80 4 rocksdb: Options.max_log_file_size: 0 2017-09-26 14:21:27.737768 7f53bdff9e80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2017-09-26 14:21:27.737769 7f53bdff9e80 4 rocksdb: Options.log_file_time_to_roll: 0 2017-09-26 14:21:27.737770 7f53bdff9e80 4 rocksdb: Options.keep_log_file_num: 1000 2017-09-26 14:21:27.737772 7f53bdff9e80 4 rocksdb: Options.recycle_log_file_num: 4 2017-09-26 14:21:27.737773 7f53bdff9e80 4 rocksdb: Options.allow_fallocate: 1 2017-09-26 14:21:27.737774 7f53bdff9e80 4 rocksdb: Options.allow_mmap_reads: 0 2017-09-26 14:21:27.737775 7f53bdff9e80 4 rocksdb: Options.allow_mmap_writes: 0 2017-09-26 14:21:27.737777 7f53bdff9e80 4 rocksdb: Options.use_direct_reads: 0 2017-09-26 14:21:27.737777 7f53bdff9e80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2017-09-26 14:21:27.737779 7f53bdff9e80 4 rocksdb: Options.create_missing_column_families: 0 2017-09-26 14:21:27.737780 7f53bdff9e80 4 rocksdb: Options.db_log_dir: 2017-09-26 14:21:27.737781 7f53bdff9e80 4 rocksdb: Options.wal_dir: db 2017-09-26 14:21:27.737782 7f53bdff9e80 4 rocksdb: Options.table_cache_numshardbits: 6 2017-09-26 14:21:27.737783 7f53bdff9e80 4 rocksdb: Options.max_subcompactions: 1 2017-09-26 14:21:27.737785 7f53bdff9e80 4 rocksdb: Options.max_background_flushes: 1 2017-09-26 14:21:27.737786 7f53bdff9e80 4 rocksdb: Options.WAL_ttl_seconds: 0 2017-09-26 14:21:27.737787 7f53bdff9e80 4 rocksdb: Options.WAL_size_limit_MB: 0 2017-09-26 14:21:27.737788 7f53bdff9e80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2017-09-26 14:21:27.737788 7f53bdff9e80 4 rocksdb: Options.is_fd_close_on_exec: 1 2017-09-26 14:21:27.737789 7f53bdff9e80 4 rocksdb: Options.advise_random_on_open: 1 2017-09-26 14:21:27.737790 7f53bdff9e80 4 rocksdb: Options.db_write_buffer_size: 0 2017-09-26 14:21:27.737791 7f53bdff9e80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2017-09-26 14:21:27.737792 7f53bdff9e80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2017-09-26 14:21:27.737793 7f53bdff9e80 4 rocksdb: Options.compaction_readahead_size: 2097152 2017-09-26 14:21:27.737794 7f53bdff9e80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2017-09-26 14:21:27.737795 7f53bdff9e80 4 rocksdb: Options.writable_file_max_buffer_size: 0 2017-09-26 14:21:27.737796 7f53bdff9e80 4 rocksdb: Options.use_adaptive_mutex: 0 2017-09-26 14:21:27.737796 7f53bdff9e80 4 rocksdb: Options.rate_limiter: (nil) 2017-09-26 14:21:27.737798 7f53bdff9e80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2017-09-26 14:21:27.737799 7f53bdff9e80 4 rocksdb: Options.bytes_per_sync: 0 2017-09-26 14:21:27.737800 7f53bdff9e80 4 rocksdb: Options.wal_bytes_per_sync: 0 2017-09-26 14:21:27.737802 7f53bdff9e80 4 rocksdb: Options.wal_recovery_mode: 0 2017-09-26 14:21:27.737803 7f53bdff9e80 4 rocksdb: Options.enable_thread_tracking: 0 2017-09-26 14:21:27.737804 7f53bdff9e80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2017-09-26 14:21:27.737806 7f53bdff9e80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2017-09-26 14:21:27.737807 7f53bdff9e80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2017-09-26 14:21:27.737808 7f53bdff9e80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2017-09-26 14:21:27.737810 7f53bdff9e80 4 rocksdb: Options.row_cache: None 2017-09-26 14:21:27.737811 7f53bdff9e80 4 rocksdb: Options.wal_filter: None 2017-09-26 14:21:27.737812 7f53bdff9e80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2017-09-26 14:21:27.737813 7f53bdff9e80 4 rocksdb: Options.base_background_compactions: 1 2017-09-26 14:21:27.737814 7f53bdff9e80 4 rocksdb: Options.max_background_compactions: 1 2017-09-26 14:21:27.737815 7f53bdff9e80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2017-09-26 14:21:27.737822 7f53bdff9e80 4 rocksdb: Options.delayed_write_rate : 16777216 2017-09-26 14:21:27.737823 7f53bdff9e80 4 rocksdb: Options.max_total_wal_size: 0 2017-09-26 14:21:27.737824 7f53bdff9e80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2017-09-26 14:21:27.737826 7f53bdff9e80 4 rocksdb: Options.stats_dump_period_sec: 600 2017-09-26 14:21:27.737827 7f53bdff9e80 4 rocksdb: Compression algorithms supported: 2017-09-26 14:21:27.737828 7f53bdff9e80 4 rocksdb: Snappy supported: 0 2017-09-26 14:21:27.737829 7f53bdff9e80 4 rocksdb: Zlib supported: 0 2017-09-26 14:21:27.737830 7f53bdff9e80 4 rocksdb: Bzip supported: 0 2017-09-26 14:21:27.737831 7f53bdff9e80 4 rocksdb: LZ4 supported: 0 2017-09-26 14:21:27.737832 7f53bdff9e80 4 rocksdb: ZSTD supported: 0 2017-09-26 14:21:27.737833 7f53bdff9e80 4 rocksdb: Fast CRC32 supported: 0 2017-09-26 14:21:27.747385 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000260 2017-09-26 14:21:27.747441 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2017-09-26 14:21:27.747444 7f53bdff9e80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2017-09-26 14:21:27.747451 7f53bdff9e80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2017-09-26 14:21:27.747457 7f53bdff9e80 4 rocksdb: Options.compaction_filter: None 2017-09-26 14:21:27.747459 7f53bdff9e80 4 rocksdb: Options.compaction_filter_factory: None 2017-09-26 14:21:27.747460 7f53bdff9e80 4 rocksdb: Options.memtable_factory: SkipListFactory 2017-09-26 14:21:27.747461 7f53bdff9e80 4 rocksdb: Options.table_factory: BlockBasedTable 2017-09-26 14:21:27.747483 7f53bdff9e80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55596864c118) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55596893b040 block_cache_name: LRUCache block_cache_options: capacity : 265751104 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 2017-09-26 14:21:27.747491 7f53bdff9e80 4 rocksdb: Options.write_buffer_size: 268435456 2017-09-26 14:21:27.747493 7f53bdff9e80 4 rocksdb: Options.max_write_buffer_number: 4 2017-09-26 14:21:27.747496 7f53bdff9e80 4 rocksdb: Options.compression: NoCompression 2017-09-26 14:21:27.747498 7f53bdff9e80 4 rocksdb: Options.bottommost_compression: Disabled 2017-09-26 14:21:27.747499 7f53bdff9e80 4 rocksdb: Options.prefix_extractor: nullptr 2017-09-26 14:21:27.747500 7f53bdff9e80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2017-09-26 14:21:27.747503 7f53bdff9e80 4 rocksdb: Options.num_levels: 7 2017-09-26 14:21:27.747504 7f53bdff9e80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2017-09-26 14:21:27.747506 7f53bdff9e80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2017-09-26 14:21:27.747507 7f53bdff9e80 4 rocksdb: Options.compression_opts.window_bits: -14 2017-09-26 14:21:27.747508 7f53bdff9e80 4 rocksdb: Options.compression_opts.level: -1 2017-09-26 14:21:27.747509 7f53bdff9e80 4 rocksdb: Options.compression_opts.strategy: 0 2017-09-26 14:21:27.747511 7f53bdff9e80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2017-09-26 14:21:27.747512 7f53bdff9e80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2017-09-26 14:21:27.747513 7f53bdff9e80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2017-09-26 14:21:27.747515 7f53bdff9e80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2017-09-26 14:21:27.747516 7f53bdff9e80 4 rocksdb: Options.target_file_size_base: 67108864 2017-09-26 14:21:27.747517 7f53bdff9e80 4 rocksdb: Options.target_file_size_multiplier: 1 2017-09-26 14:21:27.747519 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2017-09-26 14:21:27.747520 7f53bdff9e80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2017-09-26 14:21:27.747521 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2017-09-26 14:21:27.747525 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2017-09-26 14:21:27.747527 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2017-09-26 14:21:27.747528 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2017-09-26 14:21:27.747529 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2017-09-26 14:21:27.747530 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2017-09-26 14:21:27.747531 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2017-09-26 14:21:27.747533 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2017-09-26 14:21:27.747534 7f53bdff9e80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2017-09-26 14:21:27.747535 7f53bdff9e80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2017-09-26 14:21:27.747537 7f53bdff9e80 4 rocksdb: Options.arena_block_size: 33554432 2017-09-26 14:21:27.747538 7f53bdff9e80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2017-09-26 14:21:27.747539 7f53bdff9e80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2017-09-26 14:21:27.747541 7f53bdff9e80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2017-09-26 14:21:27.747544 7f53bdff9e80 4 rocksdb: Options.disable_auto_compactions: 0 2017-09-26 14:21:27.747546 7f53bdff9e80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2017-09-26 14:21:27.747548 7f53bdff9e80 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2017-09-26 14:21:27.747549 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2017-09-26 14:21:27.747552 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2017-09-26 14:21:27.747553 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2017-09-26 14:21:27.747554 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2017-09-26 14:21:27.747555 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2017-09-26 14:21:27.747556 7f53bdff9e80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2017-09-26 14:21:27.747557 7f53bdff9e80 4 rocksdb: Options.table_properties_collectors: 2017-09-26 14:21:27.747558 7f53bdff9e80 4 rocksdb: Options.inplace_update_support: 0 2017-09-26 14:21:27.747559 7f53bdff9e80 4 rocksdb: Options.inplace_update_num_locks: 10000 2017-09-26 14:21:27.747561 7f53bdff9e80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2017-09-26 14:21:27.747562 7f53bdff9e80 4 rocksdb: Options.memtable_huge_page_size: 0 2017-09-26 14:21:27.747563 7f53bdff9e80 4 rocksdb: Options.bloom_locality: 0 2017-09-26 14:21:27.747564 7f53bdff9e80 4 rocksdb: Options.max_successive_merges: 0 2017-09-26 14:21:27.747565 7f53bdff9e80 4 rocksdb: Options.optimize_filters_for_hits: 0 2017-09-26 14:21:27.747566 7f53bdff9e80 4 rocksdb: Options.paranoid_file_checks: 0 2017-09-26 14:21:27.747567 7f53bdff9e80 4 rocksdb: Options.force_consistency_checks: 0 2017-09-26 14:21:27.747569 7f53bdff9e80 4 rocksdb: Options.report_bg_io_stats: 0 2017-09-26 14:21:27.956714 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000260 succeeded,manifest_file_number is 260, next_file_number is 384, last_sequence is 42462057, log_number is 0,prev_log_number is 0,max_column_family is 0 2017-09-26 14:21:27.956735 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 381 2017-09-26 14:21:27.956860 7f53bdff9e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1506428487956848, "job": 1, "event": "recovery_started", "log_files": [361, 381]} 2017-09-26 14:21:27.956867 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #361 mode 0 2017-09-26 14:21:31.752186 7f53bdff9e80 -1 /var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7f53bdff9e80 time 2017-09-26 14:21:31.746984 /var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/os/bluestore/BlueFS.cc: 967: FAILED assert(r == 0) ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55595d7bb592] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xf3b) [0x55595d728c7b] 3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x34) [0x55595d74e084] 4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x70) [0x55595db790c0] 5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xe0) [0x55595dad6b30] 6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x55595dad6c04] 7: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::__cxx11::basic_string, std::allocator >*, rocksdb::WALRecoveryMode)+0xda) [0x55595dad6eca] 8: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0x141d) [0x55595dac86dd] 9: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool)+0x8c4) [0x55595daca034] 10: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0xedf) [0x55595dacb3ff] 11: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::DB**)+0x6a7) [0x55595daccc57] 12: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55595d6b9db8] 13: (BlueStore::_open_db(bool)+0xf6b) [0x55595d644e2b] 14: (BlueStore::_mount(bool)+0x29c) [0x55595d67695c] 15: (OSD::init()+0x3df) [0x55595d2084df] 16: (main()+0x2d5a) [0x55595d11cb1a] 17: (__libc_start_main()+0xf1) [0x7f53bab525d1] 18: (_start()+0x2a) [0x55595d1a69aa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -224> 2017-09-26 14:21:27.315473 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perfcounters_dump hook 0x5559686501e0 -223> 2017-09-26 14:21:27.315489 7f53bdff9e80 5 asok(0x55596867b2c0) register_command 1 hook 0x5559686501e0 -222> 2017-09-26 14:21:27.315552 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perf dump hook 0x5559686501e0 -221> 2017-09-26 14:21:27.315571 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perfcounters_schema hook 0x5559686501e0 -220> 2017-09-26 14:21:27.315589 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perf histogram dump hook 0x5559686501e0 -219> 2017-09-26 14:21:27.315609 7f53bdff9e80 5 asok(0x55596867b2c0) register_command 2 hook 0x5559686501e0 -218> 2017-09-26 14:21:27.315620 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perf schema hook 0x5559686501e0 -217> 2017-09-26 14:21:27.315627 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perf histogram schema hook 0x5559686501e0 -216> 2017-09-26 14:21:27.315686 7f53bdff9e80 5 asok(0x55596867b2c0) register_command perf reset hook 0x5559686501e0 -215> 2017-09-26 14:21:27.315694 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config show hook 0x5559686501e0 -214> 2017-09-26 14:21:27.315740 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config help hook 0x5559686501e0 -213> 2017-09-26 14:21:27.315748 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config set hook 0x5559686501e0 -212> 2017-09-26 14:21:27.315758 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config get hook 0x5559686501e0 -211> 2017-09-26 14:21:27.315765 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config diff hook 0x5559686501e0 -210> 2017-09-26 14:21:27.315771 7f53bdff9e80 5 asok(0x55596867b2c0) register_command config diff get hook 0x5559686501e0 -209> 2017-09-26 14:21:27.315783 7f53bdff9e80 5 asok(0x55596867b2c0) register_command log flush hook 0x5559686501e0 -208> 2017-09-26 14:21:27.315789 7f53bdff9e80 5 asok(0x55596867b2c0) register_command log dump hook 0x5559686501e0 -207> 2017-09-26 14:21:27.315796 7f53bdff9e80 5 asok(0x55596867b2c0) register_command log reopen hook 0x5559686501e0 -206> 2017-09-26 14:21:27.315815 7f53bdff9e80 5 asok(0x55596867b2c0) register_command dump_mempools hook 0x555968916b88 -205> 2017-09-26 14:21:27.323230 7f53bdff9e80 0 ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable), process (unknown), pid 1712100 -204> 2017-09-26 14:21:27.323292 7f53bdff9e80 5 object store type is bluestore -203> 2017-09-26 14:21:27.325259 7f53bdff9e80 -1 Public network was set, but cluster network was not set -202> 2017-09-26 14:21:27.325274 7f53bdff9e80 -1 Using public network also for cluster network -201> 2017-09-26 14:21:27.327425 7f53b98e1700 2 Event(0x555968656980 nevent=5000 time_id=1).set_owner idx=0 owner=139997572110080 -200> 2017-09-26 14:21:27.327446 7f53b90e0700 2 Event(0x555968656bc0 nevent=5000 time_id=1).set_owner idx=1 owner=139997563717376 -199> 2017-09-26 14:21:27.327493 7f53b88df700 2 Event(0x5559686562c0 nevent=5000 time_id=1).set_owner idx=2 owner=139997555324672 -198> 2017-09-26 14:21:27.327987 7f53bdff9e80 1 -- 194.249.156.130:0/0 learned_addr learned my addr 194.249.156.130:0/0 -197> 2017-09-26 14:21:27.327999 7f53bdff9e80 1 -- 194.249.156.130:6804/1712100 _finish_bind bind my_inst.addr is 194.249.156.130:6804/1712100 -196> 2017-09-26 14:21:27.328131 7f53bdff9e80 1 -- 194.249.156.130:0/0 learned_addr learned my addr 194.249.156.130:0/0 -195> 2017-09-26 14:21:27.328139 7f53bdff9e80 1 -- 194.249.156.130:6805/1712100 _finish_bind bind my_inst.addr is 194.249.156.130:6805/1712100 -194> 2017-09-26 14:21:27.328288 7f53bdff9e80 1 -- 194.249.156.130:0/0 learned_addr learned my addr 194.249.156.130:0/0 -193> 2017-09-26 14:21:27.328295 7f53bdff9e80 1 -- 194.249.156.130:6806/1712100 _finish_bind bind my_inst.addr is 194.249.156.130:6806/1712100 -192> 2017-09-26 14:21:27.328466 7f53bdff9e80 1 -- 194.249.156.130:0/0 learned_addr learned my addr 194.249.156.130:0/0 -191> 2017-09-26 14:21:27.328473 7f53bdff9e80 1 -- 194.249.156.130:6807/1712100 _finish_bind bind my_inst.addr is 194.249.156.130:6807/1712100 -190> 2017-09-26 14:21:27.328849 7f53bdff9e80 5 asok(0x55596867b2c0) init /var/run/ceph/ceph-osd.113.asok -189> 2017-09-26 14:21:27.328867 7f53bdff9e80 5 asok(0x55596867b2c0) bind_and_listen /var/run/ceph/ceph-osd.113.asok -188> 2017-09-26 14:21:27.328940 7f53bdff9e80 5 asok(0x55596867b2c0) register_command 0 hook 0x55596864c0c8 -187> 2017-09-26 14:21:27.328947 7f53bdff9e80 5 asok(0x55596867b2c0) register_command version hook 0x55596864c0c8 -186> 2017-09-26 14:21:27.328953 7f53bdff9e80 5 asok(0x55596867b2c0) register_command git_version hook 0x55596864c0c8 -185> 2017-09-26 14:21:27.328959 7f53bdff9e80 5 asok(0x55596867b2c0) register_command help hook 0x5559686504d0 -184> 2017-09-26 14:21:27.328964 7f53bdff9e80 5 asok(0x55596867b2c0) register_command get_command_descriptions hook 0x5559686504e0 -183> 2017-09-26 14:21:27.329017 7f53bdff9e80 10 monclient: build_initial_monmap -182> 2017-09-26 14:21:27.329019 7f53b78dd700 5 asok(0x55596867b2c0) entry start -181> 2017-09-26 14:21:27.337739 7f53bdff9e80 0 load: jerasure load: lrc load: isa -180> 2017-09-26 14:21:27.337768 7f53bdff9e80 5 adding auth protocol: cephx -179> 2017-09-26 14:21:27.337771 7f53bdff9e80 5 adding auth protocol: cephx -178> 2017-09-26 14:21:27.337825 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel -177> 2017-09-26 14:21:27.337837 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block -176> 2017-09-26 14:21:27.338169 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational -175> 2017-09-26 14:21:27.364552 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _set_cache_sizes cache_size 268435456 meta 0.01 kv 0.99 data 0 -174> 2017-09-26 14:21:27.364649 7f53bdff9e80 1 bdev(0x555968656d80 /var/lib/ceph/osd/ceph-113/block) close -173> 2017-09-26 14:21:27.610117 7f53bdff9e80 5 asok(0x55596867b2c0) register_command objecter_requests hook 0x555968650540 -172> 2017-09-26 14:21:27.610345 7f53bdff9e80 1 -- 194.249.156.130:6804/1712100 start start -171> 2017-09-26 14:21:27.610378 7f53bdff9e80 1 -- - start start -170> 2017-09-26 14:21:27.610380 7f53bdff9e80 1 -- - start start -169> 2017-09-26 14:21:27.610427 7f53bdff9e80 1 -- 194.249.156.130:6807/1712100 start start -168> 2017-09-26 14:21:27.610454 7f53bdff9e80 1 -- 194.249.156.130:6806/1712100 start start -167> 2017-09-26 14:21:27.610457 7f53bdff9e80 1 -- 194.249.156.130:6805/1712100 start start -166> 2017-09-26 14:21:27.610460 7f53bdff9e80 1 -- - start start -165> 2017-09-26 14:21:27.610643 7f53bdff9e80 2 osd.113 0 init /var/lib/ceph/osd/ceph-113 (looks like hdd) -164> 2017-09-26 14:21:27.610653 7f53bdff9e80 2 osd.113 0 journal /var/lib/ceph/osd/ceph-113/journal -163> 2017-09-26 14:21:27.610678 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _mount path /var/lib/ceph/osd/ceph-113 -162> 2017-09-26 14:21:27.610750 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel -161> 2017-09-26 14:21:27.610759 7f53bdff9e80 1 bdev(0x555968656fc0 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block -160> 2017-09-26 14:21:27.610980 7f53bdff9e80 1 bdev(0x555968656fc0 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational -159> 2017-09-26 14:21:27.614527 7f53bdff9e80 1 bluestore(/var/lib/ceph/osd/ceph-113) _set_cache_sizes cache_size 268435456 meta 0.01 kv 0.99 data 0 -158> 2017-09-26 14:21:27.614610 7f53bdff9e80 1 bdev create path /var/lib/ceph/osd/ceph-113/block type kernel -157> 2017-09-26 14:21:27.614618 7f53bdff9e80 1 bdev(0x555968657200 /var/lib/ceph/osd/ceph-113/block) open path /var/lib/ceph/osd/ceph-113/block -156> 2017-09-26 14:21:27.614829 7f53bdff9e80 1 bdev(0x555968657200 /var/lib/ceph/osd/ceph-113/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational -155> 2017-09-26 14:21:27.614838 7f53bdff9e80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-113/block size 1862 GB -154> 2017-09-26 14:21:27.614884 7f53bdff9e80 1 bluefs mount -153> 2017-09-26 14:21:27.737451 7f53bdff9e80 0 set rocksdb option compaction_readahead_size = 2097152 -152> 2017-09-26 14:21:27.737483 7f53bdff9e80 0 set rocksdb option compression = kNoCompression -151> 2017-09-26 14:21:27.737490 7f53bdff9e80 0 set rocksdb option max_write_buffer_number = 4 -150> 2017-09-26 14:21:27.737495 7f53bdff9e80 0 set rocksdb option min_write_buffer_number_to_merge = 1 -149> 2017-09-26 14:21:27.737498 7f53bdff9e80 0 set rocksdb option recycle_log_file_num = 4 -148> 2017-09-26 14:21:27.737502 7f53bdff9e80 0 set rocksdb option writable_file_max_buffer_size = 0 -147> 2017-09-26 14:21:27.737507 7f53bdff9e80 0 set rocksdb option write_buffer_size = 268435456 -146> 2017-09-26 14:21:27.737528 7f53bdff9e80 0 set rocksdb option compaction_readahead_size = 2097152 -145> 2017-09-26 14:21:27.737533 7f53bdff9e80 0 set rocksdb option compression = kNoCompression -144> 2017-09-26 14:21:27.737537 7f53bdff9e80 0 set rocksdb option max_write_buffer_number = 4 -143> 2017-09-26 14:21:27.737540 7f53bdff9e80 0 set rocksdb option min_write_buffer_number_to_merge = 1 -142> 2017-09-26 14:21:27.737544 7f53bdff9e80 0 set rocksdb option recycle_log_file_num = 4 -141> 2017-09-26 14:21:27.737547 7f53bdff9e80 0 set rocksdb option writable_file_max_buffer_size = 0 -140> 2017-09-26 14:21:27.737551 7f53bdff9e80 0 set rocksdb option write_buffer_size = 268435456 -139> 2017-09-26 14:21:27.737689 7f53bdff9e80 4 rocksdb: RocksDB version: 5.4.0 -138> 2017-09-26 14:21:27.737702 7f53bdff9e80 4 rocksdb: Git sha rocksdb_build_git_sha:@bac0af364a2c742ea9bb63199ddeee0edacdb2ac@ -137> 2017-09-26 14:21:27.737705 7f53bdff9e80 4 rocksdb: Compile date Sep 23 2017 -136> 2017-09-26 14:21:27.737708 7f53bdff9e80 4 rocksdb: DB SUMMARY -135> 2017-09-26 14:21:27.737737 7f53bdff9e80 4 rocksdb: CURRENT file: CURRENT -134> 2017-09-26 14:21:27.737742 7f53bdff9e80 4 rocksdb: IDENTITY file: IDENTITY -133> 2017-09-26 14:21:27.737745 7f53bdff9e80 4 rocksdb: MANIFEST file: MANIFEST-000260 size: 15810 Bytes -132> 2017-09-26 14:21:27.737748 7f53bdff9e80 4 rocksdb: SST files in db dir, Total Num: 18, files: 000313.sst 000314.sst 000317.sst 000354.sst 000364.sst 000366.sst 000369.sst 000370.sst 000371.sst -131> 2017-09-26 14:21:27.737750 7f53bdff9e80 4 rocksdb: Write Ahead Log file in db: 000361.log size: 242511746 ; 000381.log size: 243278096 ; -130> 2017-09-26 14:21:27.737753 7f53bdff9e80 4 rocksdb: Options.error_if_exists: 0 -129> 2017-09-26 14:21:27.737754 7f53bdff9e80 4 rocksdb: Options.create_if_missing: 0 -128> 2017-09-26 14:21:27.737756 7f53bdff9e80 4 rocksdb: Options.paranoid_checks: 1 -127> 2017-09-26 14:21:27.737758 7f53bdff9e80 4 rocksdb: Options.env: 0x555968a09f40 -126> 2017-09-26 14:21:27.737759 7f53bdff9e80 4 rocksdb: Options.info_log: 0x5559687165a0 -125> 2017-09-26 14:21:27.737762 7f53bdff9e80 4 rocksdb: Options.max_open_files: -1 -124> 2017-09-26 14:21:27.737764 7f53bdff9e80 4 rocksdb: Options.max_file_opening_threads: 16 -123> 2017-09-26 14:21:27.737765 7f53bdff9e80 4 rocksdb: Options.use_fsync: 0 -122> 2017-09-26 14:21:27.737766 7f53bdff9e80 4 rocksdb: Options.max_log_file_size: 0 -121> 2017-09-26 14:21:27.737768 7f53bdff9e80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -120> 2017-09-26 14:21:27.737769 7f53bdff9e80 4 rocksdb: Options.log_file_time_to_roll: 0 -119> 2017-09-26 14:21:27.737770 7f53bdff9e80 4 rocksdb: Options.keep_log_file_num: 1000 -118> 2017-09-26 14:21:27.737772 7f53bdff9e80 4 rocksdb: Options.recycle_log_file_num: 4 -117> 2017-09-26 14:21:27.737773 7f53bdff9e80 4 rocksdb: Options.allow_fallocate: 1 -116> 2017-09-26 14:21:27.737774 7f53bdff9e80 4 rocksdb: Options.allow_mmap_reads: 0 -115> 2017-09-26 14:21:27.737775 7f53bdff9e80 4 rocksdb: Options.allow_mmap_writes: 0 -114> 2017-09-26 14:21:27.737777 7f53bdff9e80 4 rocksdb: Options.use_direct_reads: 0 -113> 2017-09-26 14:21:27.737777 7f53bdff9e80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -112> 2017-09-26 14:21:27.737779 7f53bdff9e80 4 rocksdb: Options.create_missing_column_families: 0 -111> 2017-09-26 14:21:27.737780 7f53bdff9e80 4 rocksdb: Options.db_log_dir: -110> 2017-09-26 14:21:27.737781 7f53bdff9e80 4 rocksdb: Options.wal_dir: db -109> 2017-09-26 14:21:27.737782 7f53bdff9e80 4 rocksdb: Options.table_cache_numshardbits: 6 -108> 2017-09-26 14:21:27.737783 7f53bdff9e80 4 rocksdb: Options.max_subcompactions: 1 -107> 2017-09-26 14:21:27.737785 7f53bdff9e80 4 rocksdb: Options.max_background_flushes: 1 -106> 2017-09-26 14:21:27.737786 7f53bdff9e80 4 rocksdb: Options.WAL_ttl_seconds: 0 -105> 2017-09-26 14:21:27.737787 7f53bdff9e80 4 rocksdb: Options.WAL_size_limit_MB: 0 -104> 2017-09-26 14:21:27.737788 7f53bdff9e80 4 rocksdb: Options.manifest_preallocation_size: 4194304 -103> 2017-09-26 14:21:27.737788 7f53bdff9e80 4 rocksdb: Options.is_fd_close_on_exec: 1 -102> 2017-09-26 14:21:27.737789 7f53bdff9e80 4 rocksdb: Options.advise_random_on_open: 1 -101> 2017-09-26 14:21:27.737790 7f53bdff9e80 4 rocksdb: Options.db_write_buffer_size: 0 -100> 2017-09-26 14:21:27.737791 7f53bdff9e80 4 rocksdb: Options.access_hint_on_compaction_start: 1 -99> 2017-09-26 14:21:27.737792 7f53bdff9e80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -98> 2017-09-26 14:21:27.737793 7f53bdff9e80 4 rocksdb: Options.compaction_readahead_size: 2097152 -97> 2017-09-26 14:21:27.737794 7f53bdff9e80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -96> 2017-09-26 14:21:27.737795 7f53bdff9e80 4 rocksdb: Options.writable_file_max_buffer_size: 0 -95> 2017-09-26 14:21:27.737796 7f53bdff9e80 4 rocksdb: Options.use_adaptive_mutex: 0 -94> 2017-09-26 14:21:27.737796 7f53bdff9e80 4 rocksdb: Options.rate_limiter: (nil) -93> 2017-09-26 14:21:27.737798 7f53bdff9e80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -92> 2017-09-26 14:21:27.737799 7f53bdff9e80 4 rocksdb: Options.bytes_per_sync: 0 -91> 2017-09-26 14:21:27.737800 7f53bdff9e80 4 rocksdb: Options.wal_bytes_per_sync: 0 -90> 2017-09-26 14:21:27.737802 7f53bdff9e80 4 rocksdb: Options.wal_recovery_mode: 0 -89> 2017-09-26 14:21:27.737803 7f53bdff9e80 4 rocksdb: Options.enable_thread_tracking: 0 -88> 2017-09-26 14:21:27.737804 7f53bdff9e80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -87> 2017-09-26 14:21:27.737806 7f53bdff9e80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -86> 2017-09-26 14:21:27.737807 7f53bdff9e80 4 rocksdb: Options.write_thread_max_yield_usec: 100 -85> 2017-09-26 14:21:27.737808 7f53bdff9e80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -84> 2017-09-26 14:21:27.737810 7f53bdff9e80 4 rocksdb: Options.row_cache: None -83> 2017-09-26 14:21:27.737811 7f53bdff9e80 4 rocksdb: Options.wal_filter: None -82> 2017-09-26 14:21:27.737812 7f53bdff9e80 4 rocksdb: Options.avoid_flush_during_recovery: 0 -81> 2017-09-26 14:21:27.737813 7f53bdff9e80 4 rocksdb: Options.base_background_compactions: 1 -80> 2017-09-26 14:21:27.737814 7f53bdff9e80 4 rocksdb: Options.max_background_compactions: 1 -79> 2017-09-26 14:21:27.737815 7f53bdff9e80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -78> 2017-09-26 14:21:27.737822 7f53bdff9e80 4 rocksdb: Options.delayed_write_rate : 16777216 -77> 2017-09-26 14:21:27.737823 7f53bdff9e80 4 rocksdb: Options.max_total_wal_size: 0 -76> 2017-09-26 14:21:27.737824 7f53bdff9e80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -75> 2017-09-26 14:21:27.737826 7f53bdff9e80 4 rocksdb: Options.stats_dump_period_sec: 600 -74> 2017-09-26 14:21:27.737827 7f53bdff9e80 4 rocksdb: Compression algorithms supported: -73> 2017-09-26 14:21:27.737828 7f53bdff9e80 4 rocksdb: Snappy supported: 0 -72> 2017-09-26 14:21:27.737829 7f53bdff9e80 4 rocksdb: Zlib supported: 0 -71> 2017-09-26 14:21:27.737830 7f53bdff9e80 4 rocksdb: Bzip supported: 0 -70> 2017-09-26 14:21:27.737831 7f53bdff9e80 4 rocksdb: LZ4 supported: 0 -69> 2017-09-26 14:21:27.737832 7f53bdff9e80 4 rocksdb: ZSTD supported: 0 -68> 2017-09-26 14:21:27.737833 7f53bdff9e80 4 rocksdb: Fast CRC32 supported: 0 -67> 2017-09-26 14:21:27.747385 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000260 -66> 2017-09-26 14:21:27.747441 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: -65> 2017-09-26 14:21:27.747444 7f53bdff9e80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -64> 2017-09-26 14:21:27.747451 7f53bdff9e80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -63> 2017-09-26 14:21:27.747457 7f53bdff9e80 4 rocksdb: Options.compaction_filter: None -62> 2017-09-26 14:21:27.747459 7f53bdff9e80 4 rocksdb: Options.compaction_filter_factory: None -61> 2017-09-26 14:21:27.747460 7f53bdff9e80 4 rocksdb: Options.memtable_factory: SkipListFactory -60> 2017-09-26 14:21:27.747461 7f53bdff9e80 4 rocksdb: Options.table_factory: BlockBasedTable -59> 2017-09-26 14:21:27.747483 7f53bdff9e80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55596864c118) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55596893b040 block_cache_name: LRUCache block_cache_options: capacity : 265751104 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 -58> 2017-09-26 14:21:27.747491 7f53bdff9e80 4 rocksdb: Options.write_buffer_size: 268435456 -57> 2017-09-26 14:21:27.747493 7f53bdff9e80 4 rocksdb: Options.max_write_buffer_number: 4 -56> 2017-09-26 14:21:27.747496 7f53bdff9e80 4 rocksdb: Options.compression: NoCompression -55> 2017-09-26 14:21:27.747498 7f53bdff9e80 4 rocksdb: Options.bottommost_compression: Disabled -54> 2017-09-26 14:21:27.747499 7f53bdff9e80 4 rocksdb: Options.prefix_extractor: nullptr -53> 2017-09-26 14:21:27.747500 7f53bdff9e80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -52> 2017-09-26 14:21:27.747503 7f53bdff9e80 4 rocksdb: Options.num_levels: 7 -51> 2017-09-26 14:21:27.747504 7f53bdff9e80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -50> 2017-09-26 14:21:27.747506 7f53bdff9e80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -49> 2017-09-26 14:21:27.747507 7f53bdff9e80 4 rocksdb: Options.compression_opts.window_bits: -14 -48> 2017-09-26 14:21:27.747508 7f53bdff9e80 4 rocksdb: Options.compression_opts.level: -1 -47> 2017-09-26 14:21:27.747509 7f53bdff9e80 4 rocksdb: Options.compression_opts.strategy: 0 -46> 2017-09-26 14:21:27.747511 7f53bdff9e80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -45> 2017-09-26 14:21:27.747512 7f53bdff9e80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -44> 2017-09-26 14:21:27.747513 7f53bdff9e80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -43> 2017-09-26 14:21:27.747515 7f53bdff9e80 4 rocksdb: Options.level0_stop_writes_trigger: 36 -42> 2017-09-26 14:21:27.747516 7f53bdff9e80 4 rocksdb: Options.target_file_size_base: 67108864 -41> 2017-09-26 14:21:27.747517 7f53bdff9e80 4 rocksdb: Options.target_file_size_multiplier: 1 -40> 2017-09-26 14:21:27.747519 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -39> 2017-09-26 14:21:27.747520 7f53bdff9e80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -38> 2017-09-26 14:21:27.747521 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -37> 2017-09-26 14:21:27.747525 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -36> 2017-09-26 14:21:27.747527 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -35> 2017-09-26 14:21:27.747528 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -34> 2017-09-26 14:21:27.747529 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -33> 2017-09-26 14:21:27.747530 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -32> 2017-09-26 14:21:27.747531 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -31> 2017-09-26 14:21:27.747533 7f53bdff9e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -30> 2017-09-26 14:21:27.747534 7f53bdff9e80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -29> 2017-09-26 14:21:27.747535 7f53bdff9e80 4 rocksdb: Options.max_compaction_bytes: 1677721600 -28> 2017-09-26 14:21:27.747537 7f53bdff9e80 4 rocksdb: Options.arena_block_size: 33554432 -27> 2017-09-26 14:21:27.747538 7f53bdff9e80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -26> 2017-09-26 14:21:27.747539 7f53bdff9e80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -25> 2017-09-26 14:21:27.747541 7f53bdff9e80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -24> 2017-09-26 14:21:27.747544 7f53bdff9e80 4 rocksdb: Options.disable_auto_compactions: 0 -23> 2017-09-26 14:21:27.747546 7f53bdff9e80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -22> 2017-09-26 14:21:27.747548 7f53bdff9e80 4 rocksdb: Options.compaction_pri: kByCompensatedSize -21> 2017-09-26 14:21:27.747549 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -20> 2017-09-26 14:21:27.747552 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -19> 2017-09-26 14:21:27.747553 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -18> 2017-09-26 14:21:27.747554 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -17> 2017-09-26 14:21:27.747555 7f53bdff9e80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -16> 2017-09-26 14:21:27.747556 7f53bdff9e80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -15> 2017-09-26 14:21:27.747557 7f53bdff9e80 4 rocksdb: Options.table_properties_collectors: -14> 2017-09-26 14:21:27.747558 7f53bdff9e80 4 rocksdb: Options.inplace_update_support: 0 -13> 2017-09-26 14:21:27.747559 7f53bdff9e80 4 rocksdb: Options.inplace_update_num_locks: 10000 -12> 2017-09-26 14:21:27.747561 7f53bdff9e80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -11> 2017-09-26 14:21:27.747562 7f53bdff9e80 4 rocksdb: Options.memtable_huge_page_size: 0 -10> 2017-09-26 14:21:27.747563 7f53bdff9e80 4 rocksdb: Options.bloom_locality: 0 -9> 2017-09-26 14:21:27.747564 7f53bdff9e80 4 rocksdb: Options.max_successive_merges: 0 -8> 2017-09-26 14:21:27.747565 7f53bdff9e80 4 rocksdb: Options.optimize_filters_for_hits: 0 -7> 2017-09-26 14:21:27.747566 7f53bdff9e80 4 rocksdb: Options.paranoid_file_checks: 0 -6> 2017-09-26 14:21:27.747567 7f53bdff9e80 4 rocksdb: Options.force_consistency_checks: 0 -5> 2017-09-26 14:21:27.747569 7f53bdff9e80 4 rocksdb: Options.report_bg_io_stats: 0 -4> 2017-09-26 14:21:27.956714 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000260 succeeded,manifest_file_number is 260, next_file_number is 384, last_sequence is 42462057, log_number is 0,prev_log_number is 0,max_column_family is 0 -3> 2017-09-26 14:21:27.956735 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 381 -2> 2017-09-26 14:21:27.956860 7f53bdff9e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1506428487956848, "job": 1, "event": "recovery_started", "log_files": [361, 381]} -1> 2017-09-26 14:21:27.956867 7f53bdff9e80 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #361 mode 0 0> 2017-09-26 14:21:31.752186 7f53bdff9e80 -1 /var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7f53bdff9e80 time 2017-09-26 14:21:31.746984 /var/tmp/portage/sys-cluster/ceph-12.2.0-r3/work/ceph-12.2.0/src/os/bluestore/BlueFS.cc: 967: FAILED assert(r == 0) ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55595d7bb592] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xf3b) [0x55595d728c7b] 3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x34) [0x55595d74e084] 4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x70) [0x55595db790c0] 5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xe0) [0x55595dad6b30] 6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x55595dad6c04] 7: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::__cxx11::basic_string, std::allocator >*, rocksdb::WALRecoveryMode)+0xda) [0x55595dad6eca] 8: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0x141d) [0x55595dac86dd] 9: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool)+0x8c4) [0x55595daca034] 10: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0xedf) [0x55595dacb3ff] 11: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::DB**)+0x6a7) [0x55595daccc57] 12: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55595d6b9db8] 13: (BlueStore::_open_db(bool)+0xf6b) [0x55595d644e2b] 14: (BlueStore::_mount(bool)+0x29c) [0x55595d67695c] 15: (OSD::init()+0x3df) [0x55595d2084df] 16: (main()+0x2d5a) [0x55595d11cb1a] 17: (__libc_start_main()+0xf1) [0x7f53bab525d1] 18: (_start()+0x2a) [0x55595d1a69aa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.113.log --- end dump of recent events --- 2017-09-26 14:21:31.758224 7f53bdff9e80 -1 *** Caught signal (Aborted) ** in thread 7f53bdff9e80 thread_name:ceph-osd ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable) 1: (()+0x9beed4) [0x55595d773ed4] 2: (()+0x13be0) [0x7f53bb829be0] 3: (gsignal()+0x110) [0x7f53bab67010] 4: (abort()+0x17a) [0x7f53bab68cda] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55595d7bb71e] 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xf3b) [0x55595d728c7b] 7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x34) [0x55595d74e084] 8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x70) [0x55595db790c0] 9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xe0) [0x55595dad6b30] 10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x55595dad6c04] 11: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::__cxx11::basic_string, std::allocator >*, rocksdb::WALRecoveryMode)+0xda) [0x55595dad6eca] 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0x141d) [0x55595dac86dd] 13: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool)+0x8c4) [0x55595daca034] 14: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0xedf) [0x55595dacb3ff] 15: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::DB**)+0x6a7) [0x55595daccc57] 16: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55595d6b9db8] 17: (BlueStore::_open_db(bool)+0xf6b) [0x55595d644e2b] 18: (BlueStore::_mount(bool)+0x29c) [0x55595d67695c] 19: (OSD::init()+0x3df) [0x55595d2084df] 20: (main()+0x2d5a) [0x55595d11cb1a] 21: (__libc_start_main()+0xf1) [0x7f53bab525d1] 22: (_start()+0x2a) [0x55595d1a69aa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2017-09-26 14:21:31.758224 7f53bdff9e80 -1 *** Caught signal (Aborted) ** in thread 7f53bdff9e80 thread_name:ceph-osd ceph version 12.2.0-288-gb6f38c4fcd (b6f38c4fcd0643fd24644bf0d0ad249112727a5e) luminous (stable) 1: (()+0x9beed4) [0x55595d773ed4] 2: (()+0x13be0) [0x7f53bb829be0] 3: (gsignal()+0x110) [0x7f53bab67010] 4: (abort()+0x17a) [0x7f53bab68cda] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55595d7bb71e] 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xf3b) [0x55595d728c7b] 7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x34) [0x55595d74e084] 8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x70) [0x55595db790c0] 9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xe0) [0x55595dad6b30] 10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x55595dad6c04] 11: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::__cxx11::basic_string, std::allocator >*, rocksdb::WALRecoveryMode)+0xda) [0x55595dad6eca] 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0x141d) [0x55595dac86dd] 13: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool)+0x8c4) [0x55595daca034] 14: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0xedf) [0x55595dacb3ff] 15: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::DB**)+0x6a7) [0x55595daccc57] 16: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55595d6b9db8] 17: (BlueStore::_open_db(bool)+0xf6b) [0x55595d644e2b] 18: (BlueStore::_mount(bool)+0x29c) [0x55595d67695c] 19: (OSD::init()+0x3df) [0x55595d2084df] 20: (main()+0x2d5a) [0x55595d11cb1a] 21: (__libc_start_main()+0xf1) [0x7f53bab525d1] 22: (_start()+0x2a) [0x55595d1a69aa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.113.log --- end dump of recent events ---