2018-07-25 11:17:45.225 7f4884d05280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:17:45.225 7f4884d05280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 945 2018-07-25 11:17:45.235 7f4884d05280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:17:45.245 7f4884d05280 -1 asok(0x558669f6c240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:17:45.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) mkfs path /var/lib/ceph/osd/ceph-7/ 2018-07-25 11:17:45.245 7f4884d05280 -1 bluestore(/var/lib/ceph/osd/ceph-7/) _read_fsid unparsable uuid 2018-07-25 11:17:45.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) mkfs using provided fsid 7874f871-8031-41fb-9133-321a0706b586 2018-07-25 11:17:45.245 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:45.245 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:45.245 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:45.255 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:17:45.255 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:45.255 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:45.255 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:45.255 7f4884d05280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7//block size 149 GiB 2018-07-25 11:17:45.255 7f4884d05280 1 bluefs add_block_extent bdev 1 0x11e1800000~17d800000 2018-07-25 11:17:45.255 7f4884d05280 1 bluefs mkfs osd_uuid 7874f871-8031-41fb-9133-321a0706b586 2018-07-25 11:17:45.255 7f4884d05280 1 bluefs mkfs uuid 592c3ed5-6860-4f2c-81ee-3006d75600c9 2018-07-25 11:17:45.295 7f4884d05280 1 stupidalloc 0x0x558669f7fa40 shutdown 2018-07-25 11:17:45.295 7f4884d05280 1 bluefs mount 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:45.315 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: DB SUMMARY 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: SST files in db dir, Total Num: 0, files: 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Write Ahead Log file in db: 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.create_if_missing: 1 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.env: 0x55866a28bce0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.info_log: 0x55866a28b460 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:17:45.315 7f4884d05280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.write_buffer_manager: 0x55866a270ea0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.row_cache: None 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:17:45.325 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:222] Creating manifest 1 2018-07-25 11:17:45.365 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000001 2018-07-25 11:17:45.365 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:17:45.365 7f4884d05280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558669f313d0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x558669fe7210 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: Options.ttl: 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 0 2018-07-25 11:17:45.375 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 0 2018-07-25 11:17:45.395 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55866a398000 2018-07-25 11:17:45.395 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:17:45.415 7f4884d05280 1 freelist init 2018-07-25 11:17:45.445 7f4884d05280 1 freelist shutdown 2018-07-25 11:17:45.445 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2018-07-25 11:17:45.445 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2018-07-25 11:17:45.445 7f4884d05280 1 bluefs umount 2018-07-25 11:17:45.445 7f4884d05280 1 stupidalloc 0x0x558669f7fa40 shutdown 2018-07-25 11:17:45.445 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:45.755 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:46.095 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck <<>> check (shallow) start 2018-07-25 11:17:46.095 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:46.095 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:46.095 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:46.095 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:17:46.095 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:46.095 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:46.095 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:46.095 7f4884d05280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7//block size 149 GiB 2018-07-25 11:17:46.095 7f4884d05280 1 bluefs mount 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:46.125 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:46.125 7f4884d05280 1 rocksdb: do_open column families: [default] 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: DB SUMMARY 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: SST files in db dir, Total Num: 0, files: 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Write Ahead Log file in db: 000003.log size: 354 ; 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.env: 0x55866a28cf20 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.info_log: 0x55866a28bfa0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.write_buffer_manager: 0x55866a270f00 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.row_cache: None 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000001 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558669f31308) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55866a32e210 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:17:46.125 7f4884d05280 4 rocksdb: Options.ttl: 0 2018-07-25 11:17:46.135 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 0 2018-07-25 11:17:46.135 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 0 2018-07-25 11:17:46.135 7f4884d05280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535466148401, "job": 1, "event": "recovery_started", "log_files": [3]} 2018-07-25 11:17:46.135 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #3 mode 0 2018-07-25 11:17:46.155 7f4884d05280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535466170551, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1484, "table_properties": {"data_size": 386, "index_size": 28, "filter_size": 40, "raw_key_size": 262, "raw_average_key_size": 21, "raw_value_size": 106, "raw_average_value_size": 8, "num_data_blocks": 1, "num_entries": 12, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "1"}} 2018-07-25 11:17:46.155 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 5 2018-07-25 11:17:46.185 7f4884d05280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535466201872, "job": 1, "event": "recovery_finished"} 2018-07-25 11:17:46.235 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55866a398000 2018-07-25 11:17:46.235 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:17:46.235 7f4884d05280 1 freelist init 2018-07-25 11:17:46.235 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_alloc opening allocation metadata 2018-07-25 11:17:46.235 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:17:46.245 7f4884d05280 1 bluefs fsck 2018-07-25 11:17:46.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck walking object keyspace 2018-07-25 11:17:46.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck checking shared_blobs 2018-07-25 11:17:46.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck checking for stray omap data 2018-07-25 11:17:46.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck checking deferred events 2018-07-25 11:17:46.245 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck checking freelist vs allocated 2018-07-25 11:17:46.515 7f4884d05280 1 stupidalloc 0x0x558669f7fc00 shutdown 2018-07-25 11:17:46.515 7f4884d05280 1 freelist shutdown 2018-07-25 11:17:46.515 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2018-07-25 11:17:46.515 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2018-07-25 11:17:46.515 7f4884d05280 1 bluefs umount 2018-07-25 11:17:46.525 7f4884d05280 1 stupidalloc 0x0x558669f7fa40 shutdown 2018-07-25 11:17:46.525 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:46.845 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:46.905 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck <<>> with 0 errors, 0 repaired, 0 remaining in 0.810206 seconds 2018-07-25 11:17:46.915 7f4884d05280 0 bluestore(/var/lib/ceph/osd/ceph-7/) mkfs success 2018-07-25 11:17:46.915 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _mount path /var/lib/ceph/osd/ceph-7/ 2018-07-25 11:17:46.915 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:46.915 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:46.915 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:46.915 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:17:46.915 7f4884d05280 1 bdev create path /var/lib/ceph/osd/ceph-7//block type kernel 2018-07-25 11:17:46.915 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open path /var/lib/ceph/osd/ceph-7//block 2018-07-25 11:17:46.915 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:46.915 7f4884d05280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7//block size 149 GiB 2018-07-25 11:17:46.925 7f4884d05280 1 bluefs mount 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:46.945 7f4884d05280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:46.945 7f4884d05280 1 rocksdb: do_open column families: [default] 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: DB SUMMARY 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 168 Bytes 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000004.sst 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Write Ahead Log file in db: 000006.log size: 0 ; 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.env: 0x55866a28c460 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.info_log: 0x55866a28aee0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:17:46.945 7f4884d05280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.write_buffer_manager: 0x55866a270e40 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.row_cache: None 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000005 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558669f31320) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x558669fe7210 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:17:46.955 7f4884d05280 4 rocksdb: Options.ttl: 0 2018-07-25 11:17:46.965 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 8, last_sequence is 12, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 3 2018-07-25 11:17:46.965 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 4 2018-07-25 11:17:46.965 7f4884d05280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535466977230, "job": 1, "event": "recovery_started", "log_files": [6]} 2018-07-25 11:17:46.965 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #6 mode 0 2018-07-25 11:17:46.965 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 8 2018-07-25 11:17:46.985 7f4884d05280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535467001851, "job": 1, "event": "recovery_finished"} 2018-07-25 11:17:47.035 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x558669ffb000 2018-07-25 11:17:47.035 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:17:47.035 7f4884d05280 1 freelist init 2018-07-25 11:17:47.035 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_alloc opening allocation metadata 2018-07-25 11:17:47.035 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:17:47.105 7f4884d05280 1 bluestore(/var/lib/ceph/osd/ceph-7/) umount 2018-07-25 11:17:47.145 7f4884d05280 1 stupidalloc 0x0x558669f7fc00 shutdown 2018-07-25 11:17:47.145 7f4884d05280 1 freelist shutdown 2018-07-25 11:17:47.145 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2018-07-25 11:17:47.145 7f4884d05280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2018-07-25 11:17:47.145 7f4884d05280 1 bluefs umount 2018-07-25 11:17:47.145 7f4884d05280 1 stupidalloc 0x0x558669f7fa40 shutdown 2018-07-25 11:17:47.145 7f4884d05280 1 bdev(0x55866a3a4380 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:47.465 7f4884d05280 1 bdev(0x55866a3a4000 /var/lib/ceph/osd/ceph-7//block) close 2018-07-25 11:17:47.685 7f4884d05280 0 created object store /var/lib/ceph/osd/ceph-7/ for osd.7 fsid 922292e4-e02a-42e3-b581-34472d56c92c 2018-07-25 11:17:57.695 7fe9634fa280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:17:57.695 7fe9634fa280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1093 2018-07-25 11:17:57.695 7fe9634fa280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:17:57.695 7fe9634fa280 -1 asok(0x5586dd7fa240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:17:57.745 7fe9634fa280 0 load: jerasure load: lrc load: isa 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:57.745 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:17:58.085 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:58.085 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:17:58.085 7fe9634fa280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:17:58.085 7fe9634fa280 1 bluefs mount 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:17:58.125 7fe9634fa280 1 rocksdb: do_open column families: [default] 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: DB SUMMARY 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 168 Bytes 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000004.sst 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Write Ahead Log file in db: 000009.log size: 4542 ; 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.env: 0x5586ddb1ecc0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.info_log: 0x5586ddb41620 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_buffer_manager: 0x5586ddb3dcb0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.row_cache: None 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000008 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586dd7c33c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5586dd875490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.ttl: 0 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 11, last_sequence is 12, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 6 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 7 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478157704, "job": 1, "event": "recovery_started", "log_files": [9]} 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #9 mode 0 2018-07-25 11:17:58.265 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478277296, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 11, "file_size": 1455, "table_properties": {"data_size": 371, "index_size": 28, "filter_size": 25, "raw_key_size": 152, "raw_average_key_size": 25, "raw_value_size": 190, "raw_average_value_size": 31, "num_data_blocks": 1, "num_entries": 6, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "2"}} 2018-07-25 11:17:58.265 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 12 2018-07-25 11:17:58.405 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478410341, "job": 1, "event": "recovery_finished"} 2018-07-25 11:17:58.535 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x5586ddc45000 2018-07-25 11:17:58.535 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:17:58.545 7fe9634fa280 1 freelist init 2018-07-25 11:17:58.545 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:17:58.545 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:17:58.545 7fe9634fa280 0 _get_class not permitted to load kvs 2018-07-25 11:17:58.545 7fe9634fa280 0 _get_class not permitted to load lua 2018-07-25 11:17:58.545 7fe9634fa280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:17:58.635 7fe9634fa280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:17:58.635 7fe9634fa280 0 _get_class not permitted to load sdk 2018-07-25 11:17:58.635 7fe9634fa280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 check_osdmap_features enabling on-disk ERASURE CODES compat feature 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 load_pgs 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:17:58.645 7fe9634fa280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:17:59.975 7fe9634fa280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:17:59.975 7fe9634fa280 1 osd.7 0 start_boot 2018-07-25 11:17:59.985 7fe93fc9e700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:17:59.995 7fe9498fd700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:17:59.995 7fe9498fd700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fe9498fd700 time 2018-07-25 11:18:00.004417 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fe95a962ca2] 2: (()+0x283e67) [0x7fe95a962e67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x5586db6d8efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5586db68cff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x5586db695d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x5586db696066] 7: (DispatchQueue::entry()+0xb92) [0x7fe95a9db4a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe95aa78dad] 9: (()+0x792a) [0x7fe958eb192a] 10: (clone()+0x3f) [0x7fe957e3c87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -457> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perfcounters_dump hook 0x5586dd7c4060 -456> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command 1 hook 0x5586dd7c4060 -455> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perf dump hook 0x5586dd7c4060 -454> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perfcounters_schema hook 0x5586dd7c4060 -453> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perf histogram dump hook 0x5586dd7c4060 -452> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command 2 hook 0x5586dd7c4060 -451> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perf schema hook 0x5586dd7c4060 -450> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perf histogram schema hook 0x5586dd7c4060 -449> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command perf reset hook 0x5586dd7c4060 -448> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config show hook 0x5586dd7c4060 -447> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config help hook 0x5586dd7c4060 -446> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config set hook 0x5586dd7c4060 -445> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config unset hook 0x5586dd7c4060 -444> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config get hook 0x5586dd7c4060 -443> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config diff hook 0x5586dd7c4060 -442> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command config diff get hook 0x5586dd7c4060 -441> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command log flush hook 0x5586dd7c4060 -440> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command log dump hook 0x5586dd7c4060 -439> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command log reopen hook 0x5586dd7c4060 -438> 2018-07-25 11:17:57.635 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_mempools hook 0x5586ddb46068 -437> 2018-07-25 11:17:57.655 7fe9634fa280 10 monclient: get_monmap_and_config -436> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient: build_initial_monmap -435> 2018-07-25 11:17:57.665 7fe95578d700 2 Event(0x5586dd7fb040 nevent=5000 time_id=1).set_owner idx=0 owner=140640138090240 -434> 2018-07-25 11:17:57.665 7fe954f8c700 2 Event(0x5586dd7fb280 nevent=5000 time_id=1).set_owner idx=1 owner=140640129697536 -433> 2018-07-25 11:17:57.665 7fe95478b700 2 Event(0x5586dd7fb4c0 nevent=5000 time_id=1).set_owner idx=2 owner=140640121304832 -432> 2018-07-25 11:17:57.665 7fe9634fa280 1 Processor -- start -431> 2018-07-25 11:17:57.665 7fe9634fa280 1 -- - start start -430> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient: init -429> 2018-07-25 11:17:57.665 7fe9634fa280 5 adding auth protocol: cephx -428> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient: auth_supported 2 method cephx -427> 2018-07-25 11:17:57.665 7fe9634fa280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -426> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient: _reopen_session rank -1 -425> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient(hunting): picked mon.mimir con 0x5586ddc24700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -424> 2018-07-25 11:17:57.665 7fe9634fa280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5586ddb94780 con 0 -423> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient(hunting): _renew_subs -422> 2018-07-25 11:17:57.665 7fe9634fa280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:22:57.681159 -421> 2018-07-25 11:17:57.675 7fe954f8c700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 -420> 2018-07-25 11:17:57.675 7fe954f8c700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -419> 2018-07-25 11:17:57.675 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62448 cs=1 l=1). rx mon.2 seq 1 0x5586ddc36000 mon_map magic: 0 v1 -418> 2018-07-25 11:17:57.675 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62448 cs=1 l=1). rx mon.2 seq 2 0x5586ddb94780 auth_reply(proto 2 0 (0) Success) v1 -417> 2018-07-25 11:17:57.675 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5586ddc36000 con 0x5586ddc24700 -416> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -415> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -414> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -413> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2867951144 0 0) 0x5586ddb94780 con 0x5586ddc24700 -412> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient(hunting): my global_id is 220340 -411> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5586ddb94f00 con 0 -410> 2018-07-25 11:17:57.685 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62448 cs=1 l=1). rx mon.2 seq 3 0x5586ddb94f00 auth_reply(proto 2 0 (0) Success) v1 -409> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3556763626 0 0) 0x5586ddb94f00 con 0x5586ddc24700 -408> 2018-07-25 11:17:57.685 7fe953f8a700 1 monclient: found mon.mimir -407> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -406> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586dd7fb680 con 0 -405> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:17:27.696790) -404> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -403> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5586ddb94780 con 0 -402> 2018-07-25 11:17:57.685 7fe9634fa280 5 monclient: authenticate success, global_id 220340 -401> 2018-07-25 11:17:57.685 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62448 cs=1 l=1). rx mon.2 seq 4 0x5586dd7fb680 config(0 keys) v1 -400> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x5586dd7fb680 con 0x5586ddc24700 -399> 2018-07-25 11:17:57.685 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62448 cs=1 l=1). rx mon.2 seq 5 0x5586ddc36480 mon_map magic: 0 v1 -398> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: handle_config config(0 keys) v1 -397> 2018-07-25 11:17:57.685 7fe953f8a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5586ddc36480 con 0x5586ddc24700 -396> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: handle_monmap mon_map magic: 0 v1 -395> 2018-07-25 11:17:57.685 7fe952787700 4 set_mon_vals no callback set -394> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: got monmap 3, mon.mimir is now rank 2 -393> 2018-07-25 11:17:57.685 7fe953f8a700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -392> 2018-07-25 11:17:57.685 7fe9634fa280 10 monclient: get_monmap_and_config success -391> 2018-07-25 11:17:57.685 7fe9634fa280 10 monclient: shutdown -390> 2018-07-25 11:17:57.685 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN pgs=62448 cs=1 l=1).mark_down -389> 2018-07-25 11:17:57.685 7fe9634fa280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc24700 :-1 s=STATE_OPEN pgs=62448 cs=1 l=1)._stop -388> 2018-07-25 11:17:57.685 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 shutdown_connections -387> 2018-07-25 11:17:57.685 7fe9634fa280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x5586ddc24700 -386> 2018-07-25 11:17:57.685 7fe9634fa280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 shutdown_connections delete 0x5586ddc24700 -385> 2018-07-25 11:17:57.685 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 shutdown_connections -384> 2018-07-25 11:17:57.685 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 wait complete. -383> 2018-07-25 11:17:57.685 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 conn(0x5586ddc24000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -382> 2018-07-25 11:17:57.685 7fe9634fa280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/1435224510 conn(0x5586ddc24000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -381> 2018-07-25 11:17:57.695 7fe9634fa280 0 set uid:gid to 114:993 (ceph:ceph) -380> 2018-07-25 11:17:57.695 7fe9634fa280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1093 -379> 2018-07-25 11:17:57.695 7fe9634fa280 0 pidfile_write: ignore empty --pid-file -378> 2018-07-25 11:17:57.695 7fe9634fa280 5 asok(0x5586dd7fa240) init /var/run/ceph/ceph-osd.7.asok -377> 2018-07-25 11:17:57.695 7fe9634fa280 5 asok(0x5586dd7fa240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -376> 2018-07-25 11:17:57.695 7fe9634fa280 -1 asok(0x5586dd7fa240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -375> 2018-07-25 11:17:57.695 7fe9634fa280 5 object store type is bluestore -374> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -373> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 -372> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -371> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1093 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1093 -370> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -369> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1093 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1093 -368> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -367> 2018-07-25 11:17:57.725 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1093 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1093 -366> 2018-07-25 11:17:57.725 7fe9634fa280 10 monclient: build_initial_monmap -365> 2018-07-25 11:17:57.745 7fe9634fa280 0 load: jerasure load: lrc load: isa -364> 2018-07-25 11:17:57.745 7fe9634fa280 5 adding auth protocol: cephx -363> 2018-07-25 11:17:57.745 7fe9634fa280 5 adding auth protocol: cephx -362> 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -361> 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -360> 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -359> 2018-07-25 11:17:57.745 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -358> 2018-07-25 11:17:57.745 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) close -357> 2018-07-25 11:17:58.085 7fe9634fa280 5 asok(0x5586dd7fa240) register_command objecter_requests hook 0x5586dd7c4680 -356> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 start start -355> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- - start start -354> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- - start start -353> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1093 start start -352> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1093 start start -351> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1093 start start -350> 2018-07-25 11:17:58.085 7fe9634fa280 1 -- - start start -349> 2018-07-25 11:17:58.085 7fe9634fa280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -348> 2018-07-25 11:17:58.085 7fe9634fa280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -347> 2018-07-25 11:17:58.085 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -346> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -345> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -344> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -343> 2018-07-25 11:17:58.085 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -342> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -341> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -340> 2018-07-25 11:17:58.085 7fe9634fa280 1 bdev(0x5586ddcf8380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -339> 2018-07-25 11:17:58.085 7fe9634fa280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -338> 2018-07-25 11:17:58.085 7fe9634fa280 1 bluefs mount -337> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compaction_readahead_size = 2097152 -336> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compression = kNoCompression -335> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option max_write_buffer_number = 4 -334> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -333> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option recycle_log_file_num = 4 -332> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option writable_file_max_buffer_size = 0 -331> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option write_buffer_size = 268435456 -330> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compaction_readahead_size = 2097152 -329> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option compression = kNoCompression -328> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option max_write_buffer_number = 4 -327> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -326> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option recycle_log_file_num = 4 -325> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option writable_file_max_buffer_size = 0 -324> 2018-07-25 11:17:58.115 7fe9634fa280 0 set rocksdb option write_buffer_size = 268435456 -323> 2018-07-25 11:17:58.125 7fe9634fa280 1 rocksdb: do_open column families: [default] -322> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: RocksDB version: 5.13.0 -321> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -320> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Compile date Jul 24 2018 -319> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: DB SUMMARY -318> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: CURRENT file: CURRENT -317> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: IDENTITY file: IDENTITY -316> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 168 Bytes -315> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000004.sst -314> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Write Ahead Log file in db: 000009.log size: 4542 ; -313> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.error_if_exists: 0 -312> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.create_if_missing: 0 -311> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.paranoid_checks: 1 -310> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.env: 0x5586ddb1ecc0 -309> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.info_log: 0x5586ddb41620 -308> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_file_opening_threads: 16 -307> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.statistics: (nil) -306> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_fsync: 0 -305> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_log_file_size: 0 -304> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -303> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.log_file_time_to_roll: 0 -302> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.keep_log_file_num: 1000 -301> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.recycle_log_file_num: 4 -300> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_fallocate: 1 -299> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_mmap_reads: 0 -298> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_mmap_writes: 0 -297> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_direct_reads: 0 -296> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -295> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.create_missing_column_families: 0 -294> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.db_log_dir: -293> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_dir: db -292> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_cache_numshardbits: 6 -291> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_subcompactions: 1 -290> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_flushes: -1 -289> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.WAL_ttl_seconds: 0 -288> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.WAL_size_limit_MB: 0 -287> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -286> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.is_fd_close_on_exec: 1 -285> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.advise_random_on_open: 1 -284> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.db_write_buffer_size: 0 -283> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_buffer_manager: 0x5586ddb3dcb0 -282> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -281> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -280> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -279> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.use_adaptive_mutex: 0 -278> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.rate_limiter: (nil) -277> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -276> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_recovery_mode: 0 -275> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_thread_tracking: 0 -274> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_pipelined_write: 0 -273> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -272> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -271> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -270> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -269> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.row_cache: None -268> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_filter: None -267> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -266> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.allow_ingest_behind: 0 -265> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.preserve_deletes: 0 -264> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.two_write_queues: 0 -263> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.manual_wal_flush: 0 -262> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_jobs: 2 -261> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_background_compactions: -1 -260> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -259> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -258> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.delayed_write_rate : 16777216 -257> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_total_wal_size: 0 -256> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -255> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.stats_dump_period_sec: 600 -254> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_open_files: -1 -253> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bytes_per_sync: 0 -252> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.wal_bytes_per_sync: 0 -251> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_readahead_size: 2097152 -250> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Compression algorithms supported: -249> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -248> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZSTD supported: 0 -247> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kXpressCompression supported: 0 -246> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kLZ4HCCompression supported: 1 -245> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kLZ4Compression supported: 1 -244> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kBZip2Compression supported: 0 -243> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kZlibCompression supported: 1 -242> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: kSnappyCompression supported: 1 -241> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -240> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000008 -239> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -238> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -237> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -236> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_filter: None -235> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_filter_factory: None -234> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_factory: SkipListFactory -233> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_factory: BlockBasedTable -232> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586dd7c33c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5586dd875490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -231> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.write_buffer_size: 268435456 -230> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_write_buffer_number: 4 -229> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression: NoCompression -228> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bottommost_compression: Disabled -227> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.prefix_extractor: nullptr -226> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -225> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.num_levels: 7 -224> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -223> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -222> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.window_bits: -14 -221> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.level: -1 -220> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.strategy: 0 -219> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -218> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -217> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -216> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -215> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -214> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.target_file_size_base: 67108864 -213> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.target_file_size_multiplier: 1 -212> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -211> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -210> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -209> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -208> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -207> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -206> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -205> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -204> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -203> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -202> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -201> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -200> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.arena_block_size: 33554432 -199> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -198> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -197> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -196> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.disable_auto_compactions: 0 -195> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -194> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -193> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -192> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -191> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -190> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -189> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -188> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -187> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -186> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -185> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -184> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.table_properties_collectors: -183> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.inplace_update_support: 0 -182> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.inplace_update_num_locks: 10000 -181> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -180> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.memtable_huge_page_size: 0 -179> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.bloom_locality: 0 -178> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.max_successive_merges: 0 -177> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.optimize_filters_for_hits: 0 -176> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.paranoid_file_checks: 0 -175> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.force_consistency_checks: 0 -174> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.report_bg_io_stats: 0 -173> 2018-07-25 11:17:58.135 7fe9634fa280 4 rocksdb: Options.ttl: 0 -172> 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 11, last_sequence is 12, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 6 -171> 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 7 -170> 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478157704, "job": 1, "event": "recovery_started", "log_files": [9]} -169> 2018-07-25 11:17:58.145 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #9 mode 0 -168> 2018-07-25 11:17:58.155 7fe9634fa280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:919] [default] [WriteLevel0TableForRecovery] Level-0 table #11: started -167> 2018-07-25 11:17:58.265 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478277296, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 11, "file_size": 1455, "table_properties": {"data_size": 371, "index_size": 28, "filter_size": 25, "raw_key_size": 152, "raw_average_key_size": 25, "raw_value_size": 190, "raw_average_value_size": 31, "num_data_blocks": 1, "num_entries": 6, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "2"}} -166> 2018-07-25 11:17:58.265 7fe9634fa280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:959] [default] [WriteLevel0TableForRecovery] Level-0 table #11: 1455 bytes OK -165> 2018-07-25 11:17:58.265 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 12 -164> 2018-07-25 11:17:58.405 7fe9634fa280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535478410341, "job": 1, "event": "recovery_finished"} -163> 2018-07-25 11:17:58.405 7fe9634fa280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000008 type=3 #8 -- OK -162> 2018-07-25 11:17:58.475 7fe9634fa280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000009.log type=0 #9 -- OK -161> 2018-07-25 11:17:58.535 7fe9634fa280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x5586ddc45000 -160> 2018-07-25 11:17:58.535 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -159> 2018-07-25 11:17:58.545 7fe9634fa280 1 freelist init -158> 2018-07-25 11:17:58.545 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -157> 2018-07-25 11:17:58.545 7fe9634fa280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -156> 2018-07-25 11:17:58.545 7fe9634fa280 2 osd.7 0 journal looks like hdd -155> 2018-07-25 11:17:58.545 7fe9634fa280 2 osd.7 0 boot -154> 2018-07-25 11:17:58.545 7fe9634fa280 0 _get_class not permitted to load kvs -153> 2018-07-25 11:17:58.545 7fe9634fa280 0 _get_class not permitted to load lua -152> 2018-07-25 11:17:58.545 7fe9634fa280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -151> 2018-07-25 11:17:58.565 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -150> 2018-07-25 11:17:58.575 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -149> 2018-07-25 11:17:58.575 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -148> 2018-07-25 11:17:58.575 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -147> 2018-07-25 11:17:58.585 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -146> 2018-07-25 11:17:58.585 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -145> 2018-07-25 11:17:58.585 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -144> 2018-07-25 11:17:58.615 7fe9634fa280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -143> 2018-07-25 11:17:58.635 7fe9634fa280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -142> 2018-07-25 11:17:58.635 7fe9634fa280 0 _get_class not permitted to load sdk -141> 2018-07-25 11:17:58.635 7fe9634fa280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -140> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -139> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -138> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -137> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 check_osdmap_features enabling on-disk ERASURE CODES compat feature -136> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 load_pgs -135> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 load_pgs opened 0 pgs -134> 2018-07-25 11:17:58.635 7fe9634fa280 2 osd.7 0 superblock: I am osd.7 -133> 2018-07-25 11:17:58.635 7fe9634fa280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -132> 2018-07-25 11:17:58.635 7fe9634fa280 1 Processor -- start -131> 2018-07-25 11:17:58.635 7fe9634fa280 1 Processor -- start -130> 2018-07-25 11:17:58.635 7fe9634fa280 1 Processor -- start -129> 2018-07-25 11:17:58.645 7fe9634fa280 1 Processor -- start -128> 2018-07-25 11:17:58.645 7fe9634fa280 1 Processor -- start -127> 2018-07-25 11:17:58.645 7fe9634fa280 1 Processor -- start -126> 2018-07-25 11:17:58.645 7fe9634fa280 1 Processor -- start -125> 2018-07-25 11:17:58.645 7fe9634fa280 10 monclient: init -124> 2018-07-25 11:17:58.645 7fe9634fa280 5 adding auth protocol: cephx -123> 2018-07-25 11:17:58.645 7fe9634fa280 10 monclient: auth_supported 2 method cephx -122> 2018-07-25 11:17:58.645 7fe9634fa280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -121> 2018-07-25 11:17:58.645 7fe9634fa280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -120> 2018-07-25 11:17:58.645 7fe9634fa280 -1 osd.7 0 log_to_monitors {default=true} -119> 2018-07-25 11:17:58.645 7fe9634fa280 10 monclient: _reopen_session rank -1 -118> 2018-07-25 11:17:58.645 7fe9634fa280 10 monclient(hunting): picked mon.mimir con 0x5586ddc27800 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -117> 2018-07-25 11:17:58.645 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5586ddb95900 con 0 -116> 2018-07-25 11:17:58.645 7fe9634fa280 10 monclient(hunting): _renew_subs -115> 2018-07-25 11:17:58.645 7fe93848f700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -114> 2018-07-25 11:17:58.645 7fe954f8c700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -113> 2018-07-25 11:17:58.645 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 1 0x5586ddc36900 mon_map magic: 0 v1 -112> 2018-07-25 11:17:58.645 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 2 0x5586ddb94780 auth_reply(proto 2 0 (0) Success) v1 -111> 2018-07-25 11:17:58.645 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5586ddc36900 con 0x5586ddc27800 -110> 2018-07-25 11:17:58.645 7fe9498fd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -109> 2018-07-25 11:17:58.645 7fe9498fd700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -108> 2018-07-25 11:17:58.645 7fe9498fd700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -107> 2018-07-25 11:17:58.645 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1111842531 0 0) 0x5586ddb94780 con 0x5586ddc27800 -106> 2018-07-25 11:17:58.645 7fe9498fd700 10 monclient(hunting): my global_id is 220343 -105> 2018-07-25 11:17:58.645 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5586e1e98000 con 0 -104> 2018-07-25 11:17:58.655 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 3 0x5586e1e98000 auth_reply(proto 2 0 (0) Success) v1 -103> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2548681537 0 0) 0x5586e1e98000 con 0x5586ddc27800 -102> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5586ddb94780 con 0 -101> 2018-07-25 11:17:58.655 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 4 0x5586ddb94780 auth_reply(proto 2 0 (0) Success) v1 -100> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (359680799 0 0) 0x5586ddb94780 con 0x5586ddc27800 -99> 2018-07-25 11:17:58.655 7fe9498fd700 1 monclient: found mon.mimir -98> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -97> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586dd7fbd40 con 0 -96> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:17:28.668654) -95> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -94> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5586e1e98000 con 0 -93> 2018-07-25 11:17:58.655 7fe9634fa280 5 monclient: authenticate success, global_id 220343 -92> 2018-07-25 11:17:58.655 7fe9634fa280 10 monclient: wait_auth_rotating waiting (until 2018-07-25 11:18:28.668825) -91> 2018-07-25 11:17:58.655 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 5 0x5586dd7fbd40 config(0 keys) v1 -90> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x5586dd7fbd40 con 0x5586ddc27800 -89> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: handle_config config(0 keys) v1 -88> 2018-07-25 11:17:58.655 7fe93fc9e700 4 set_mon_vals no callback set -87> 2018-07-25 11:17:58.655 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 6 0x5586ddc37b00 mon_map magic: 0 v1 -86> 2018-07-25 11:17:58.655 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 7 0x5586e1e98000 auth_reply(proto 2 0 (0) Success) v1 -85> 2018-07-25 11:17:58.655 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5586ddc37b00 con 0x5586ddc27800 -84> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: handle_monmap mon_map magic: 0 v1 -83> 2018-07-25 11:17:58.655 7fe9498fd700 10 monclient: got monmap 3, mon.mimir is now rank 2 -82> 2018-07-25 11:17:58.665 7fe9498fd700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -81> 2018-07-25 11:17:58.665 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (4145736475 0 0) 0x5586e1e98000 con 0x5586ddc27800 -80> 2018-07-25 11:17:58.665 7fe9498fd700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:17:28.674387) -79> 2018-07-25 11:17:58.665 7fe9634fa280 10 monclient: wait_auth_rotating done -78> 2018-07-25 11:17:58.665 7fe9634fa280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -77> 2018-07-25 11:17:58.665 7fe9634fa280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -76> 2018-07-25 11:17:58.665 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x5586dd7fb680 con 0 -75> 2018-07-25 11:17:58.805 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 8 0x5586dd7fb680 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 set osd(s) 7 to class 'hdd' v2918) v1 -74> 2018-07-25 11:17:58.805 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 set osd(s) 7 to class 'hdd' v2918) v1 ==== 131+0+0 (450945680 0 0) 0x5586dd7fb680 con 0x5586ddc27800 -73> 2018-07-25 11:17:58.805 7fe9498fd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -72> 2018-07-25 11:17:58.805 7fe9498fd700 10 monclient: _finish_command 1 = 0 set osd(s) 7 to class 'hdd' -71> 2018-07-25 11:17:58.805 7fe9634fa280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -70> 2018-07-25 11:17:58.805 7fe9634fa280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -69> 2018-07-25 11:17:58.805 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x5586e1ea6240 con 0 -68> 2018-07-25 11:17:59.975 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 9 0x5586e1ea6240 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updating item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2919) v1 -67> 2018-07-25 11:17:59.975 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updating item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2919) v1 ==== 246+0+0 (241423211 0 0) 0x5586e1ea6240 con 0x5586ddc27800 -66> 2018-07-25 11:17:59.975 7fe9498fd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -65> 2018-07-25 11:17:59.975 7fe9498fd700 10 monclient: _finish_command 2 = 0 create-or-move updating item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -64> 2018-07-25 11:17:59.975 7fe9634fa280 0 osd.7 0 done with init, starting boot process -63> 2018-07-25 11:17:59.975 7fe9634fa280 10 monclient: _renew_subs -62> 2018-07-25 11:17:59.975 7fe9634fa280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -61> 2018-07-25 11:17:59.975 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x5586ddc36240 con 0 -60> 2018-07-25 11:17:59.975 7fe9634fa280 1 osd.7 0 start_boot -59> 2018-07-25 11:17:59.975 7fe9634fa280 10 monclient: get_version osdmap req 0x5586e1e8adc0 -58> 2018-07-25 11:17:59.975 7fe9634fa280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -57> 2018-07-25 11:17:59.975 7fe9634fa280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x5586ddd21000 con 0 -56> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command status hook 0x5586dd7c4910 -55> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command flush_journal hook 0x5586dd7c4910 -54> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_ops_in_flight hook 0x5586dd7c4910 -53> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command ops hook 0x5586dd7c4910 -52> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_blocked_ops hook 0x5586dd7c4910 -51> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_historic_ops hook 0x5586dd7c4910 -50> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_historic_slow_ops hook 0x5586dd7c4910 -49> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_historic_ops_by_duration hook 0x5586dd7c4910 -48> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_op_pq_state hook 0x5586dd7c4910 -47> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_blacklist hook 0x5586dd7c4910 -46> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_watchers hook 0x5586dd7c4910 -45> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_reservations hook 0x5586dd7c4910 -44> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command get_latest_osdmap hook 0x5586dd7c4910 -43> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command heap hook 0x5586dd7c4910 -42> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command set_heap_property hook 0x5586dd7c4910 -41> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command get_heap_property hook 0x5586dd7c4910 -40> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_objectstore_kv_stats hook 0x5586dd7c4910 -39> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_scrubs hook 0x5586dd7c4910 -38> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command calc_objectstore_db_histogram hook 0x5586dd7c4910 -37> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command flush_store_cache hook 0x5586dd7c4910 -36> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command dump_pgstate_history hook 0x5586dd7c4910 -35> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command compact hook 0x5586dd7c4910 -34> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command get_mapped_pools hook 0x5586dd7c4910 -33> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command smart hook 0x5586dd7c4910 -32> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command list_devices hook 0x5586dd7c4910 -31> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command setomapval hook 0x5586e1e8b100 -30> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command rmomapkey hook 0x5586e1e8b100 -29> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command setomapheader hook 0x5586e1e8b100 -28> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command getomap hook 0x5586e1e8b100 -27> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command truncobj hook 0x5586e1e8b100 -26> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command injectdataerr hook 0x5586e1e8b100 -25> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command injectmdataerr hook 0x5586e1e8b100 -24> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command set_recovery_delay hook 0x5586e1e8b100 -23> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command trigger_scrub hook 0x5586e1e8b100 -22> 2018-07-25 11:17:59.975 7fe9634fa280 5 asok(0x5586dd7fa240) register_command injectfull hook 0x5586e1e8b100 -21> 2018-07-25 11:17:59.975 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 10 0x5586dfe63800 mgrmap(e 34) v1 -20> 2018-07-25 11:17:59.975 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 11 0x5586ddc30000 mon_get_version_reply(handle=1 version=2919) v2 -19> 2018-07-25 11:17:59.975 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x5586dfe63800 con 0x5586ddc27800 -18> 2018-07-25 11:17:59.975 7fe9498fd700 4 mgrc handle_mgr_map Got map version 34 -17> 2018-07-25 11:17:59.975 7fe9498fd700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -16> 2018-07-25 11:17:59.975 7fe9498fd700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -15> 2018-07-25 11:17:59.985 7fe95478b700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x5586ddf58000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -14> 2018-07-25 11:17:59.985 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x5586ddcf8a80 con 0 -13> 2018-07-25 11:17:59.985 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2919) v2 ==== 24+0+0 (536566089 0 0) 0x5586ddc30000 con 0x5586ddc27800 -12> 2018-07-25 11:17:59.985 7fe9498fd700 10 monclient: handle_get_version_reply finishing 0x5586e1e8adc0 version 2919 -11> 2018-07-25 11:17:59.985 7fe93fc9e700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -10> 2018-07-25 11:17:59.985 7fe93fc9e700 -1 osd.7 0 waiting for initial osdmap -9> 2018-07-25 11:17:59.985 7fe93fc9e700 10 monclient: _renew_subs -8> 2018-07-25 11:17:59.985 7fe93fc9e700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -7> 2018-07-25 11:17:59.985 7fe93fc9e700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x5586e1ea7200 con 0 -6> 2018-07-25 11:17:59.985 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 12 0x5586ddf7a000 osd_map(2319..2319 src has 2319..2919 +gap_removed_snaps) v4 -5> 2018-07-25 11:17:59.985 7fe9498fd700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2919 +gap_removed_snaps) v4 ==== 4888+0+0 (2187507414 0 0) 0x5586ddf7a000 con 0x5586ddc27800 -4> 2018-07-25 11:17:59.985 7fe9498fd700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2919] -3> 2018-07-25 11:17:59.995 7fe9498fd700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes -2> 2018-07-25 11:17:59.995 7fe954f8c700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5586ddc27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62449 cs=1 l=1). rx mon.2 seq 13 0x5586ddf7a2c0 osd_map(2320..2359 src has 2319..2919) v4 -1> 2018-07-25 11:17:59.995 7fe95478b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1093 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x5586ddf58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339629 cs=1 l=1). rx mgr.34109 seq 1 0x5586ddf98000 mgrconfigure(period=5, threshold=5) v2 0> 2018-07-25 11:17:59.995 7fe9498fd700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fe9498fd700 time 2018-07-25 11:18:00.004417 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fe95a962ca2] 2: (()+0x283e67) [0x7fe95a962e67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x5586db6d8efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5586db68cff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x5586db695d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x5586db696066] 7: (DispatchQueue::entry()+0xb92) [0x7fe95a9db4a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe95aa78dad] 9: (()+0x792a) [0x7fe958eb192a] 10: (clone()+0x3f) [0x7fe957e3c87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:18:00.025 7fe9498fd700 -1 *** Caught signal (Aborted) ** in thread 7fe9498fd700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x5586dbba8120] 2: (()+0x14120) [0x7fe958ebe120] 3: (gsignal()+0x110) [0x7fe957d6ffc0] 4: (abort()+0x1c7) [0x7fe957d71b07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fe95a962df0] 6: (()+0x283e67) [0x7fe95a962e67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x5586db6d8efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5586db68cff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x5586db695d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x5586db696066] 11: (DispatchQueue::entry()+0xb92) [0x7fe95a9db4a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe95aa78dad] 13: (()+0x792a) [0x7fe958eb192a] 14: (clone()+0x3f) [0x7fe957e3c87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:18:00.025 7fe9498fd700 -1 *** Caught signal (Aborted) ** in thread 7fe9498fd700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x5586dbba8120] 2: (()+0x14120) [0x7fe958ebe120] 3: (gsignal()+0x110) [0x7fe957d6ffc0] 4: (abort()+0x1c7) [0x7fe957d71b07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fe95a962df0] 6: (()+0x283e67) [0x7fe95a962e67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x5586db6d8efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5586db68cff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x5586db695d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x5586db696066] 11: (DispatchQueue::entry()+0xb92) [0x7fe95a9db4a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe95aa78dad] 13: (()+0x792a) [0x7fe958eb192a] 14: (clone()+0x3f) [0x7fe957e3c87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:18:24.216 7f395d956280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:18:24.216 7f395d956280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1190 2018-07-25 11:18:24.216 7f395d956280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:18:24.216 7f395d956280 -1 asok(0x56137c9b0240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:18:24.256 7f395d956280 0 load: jerasure load: lrc load: isa 2018-07-25 11:18:24.256 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:24.256 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:18:24.586 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:18:24.586 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:24.586 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:18:24.586 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:24.586 7f395d956280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:18:24.586 7f395d956280 1 bluefs mount 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:18:24.626 7f395d956280 1 rocksdb: do_open column families: [default] 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: DB SUMMARY 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: MANIFEST file: MANIFEST-000012 size: 209 Bytes 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: SST files in db dir, Total Num: 2, files: 000004.sst 000011.sst 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Write Ahead Log file in db: 000013.log size: 4582 ; 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.env: 0x56137ccd53e0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.info_log: 0x56137ccf7600 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_buffer_manager: 0x56137ccf3c20 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.row_cache: None 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000012 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56137c9793c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56137ca2b490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.ttl: 0 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000012 succeeded,manifest_file_number is 12, next_file_number is 15, last_sequence is 20, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 9 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 10 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504692284, "job": 1, "event": "recovery_started", "log_files": [13]} 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #13 mode 0 2018-07-25 11:18:24.756 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504767930, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 15, "file_size": 5670, "table_properties": {"data_size": 4551, "index_size": 48, "filter_size": 40, "raw_key_size": 191, "raw_average_key_size": 31, "raw_value_size": 4328, "raw_average_value_size": 721, "num_data_blocks": 2, "num_entries": 6, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "1"}} 2018-07-25 11:18:24.756 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 16 2018-07-25 11:18:24.886 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504901105, "job": 1, "event": "recovery_finished"} 2018-07-25 11:18:25.026 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x56137cdfd000 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:18:25.026 7f395d956280 1 freelist init 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:18:25.066 7f395d956280 0 _get_class not permitted to load kvs 2018-07-25 11:18:25.066 7f395d956280 0 _get_class not permitted to load lua 2018-07-25 11:18:25.066 7f395d956280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:18:25.096 7f395d956280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:18:25.096 7f395d956280 0 _get_class not permitted to load sdk 2018-07-25 11:18:25.096 7f395d956280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 load_pgs 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:18:25.096 7f395d956280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:18:25.216 7f395d956280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:18:25.216 7f395d956280 1 osd.7 0 start_boot 2018-07-25 11:18:25.226 7f393a0fa700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:18:25.236 7f3943d59700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:18:25.246 7f3943d59700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3943d59700 time 2018-07-25 11:18:25.246891 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3954dbeca2] 2: (()+0x283e67) [0x7f3954dbee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x56137a11fefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x56137a0d3ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x56137a0dcd41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x56137a0dd066] 7: (DispatchQueue::entry()+0xb92) [0x7f3954e374a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3954ed4dad] 9: (()+0x792a) [0x7f395330d92a] 10: (clone()+0x3f) [0x7f395229887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -463> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perfcounters_dump hook 0x56137c97a060 -462> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command 1 hook 0x56137c97a060 -461> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perf dump hook 0x56137c97a060 -460> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perfcounters_schema hook 0x56137c97a060 -459> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perf histogram dump hook 0x56137c97a060 -458> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command 2 hook 0x56137c97a060 -457> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perf schema hook 0x56137c97a060 -456> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perf histogram schema hook 0x56137c97a060 -455> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command perf reset hook 0x56137c97a060 -454> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config show hook 0x56137c97a060 -453> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config help hook 0x56137c97a060 -452> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config set hook 0x56137c97a060 -451> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config unset hook 0x56137c97a060 -450> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config get hook 0x56137c97a060 -449> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config diff hook 0x56137c97a060 -448> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command config diff get hook 0x56137c97a060 -447> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command log flush hook 0x56137c97a060 -446> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command log dump hook 0x56137c97a060 -445> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command log reopen hook 0x56137c97a060 -444> 2018-07-25 11:18:24.156 7f395d956280 5 asok(0x56137c9b0240) register_command dump_mempools hook 0x56137ccfc068 -443> 2018-07-25 11:18:24.186 7f395d956280 10 monclient: get_monmap_and_config -442> 2018-07-25 11:18:24.186 7f395d956280 10 monclient: build_initial_monmap -441> 2018-07-25 11:18:24.196 7f394ebe7700 2 Event(0x56137c9b14c0 nevent=5000 time_id=1).set_owner idx=2 owner=139884110968576 -440> 2018-07-25 11:18:24.196 7f394f3e8700 2 Event(0x56137c9b1280 nevent=5000 time_id=1).set_owner idx=1 owner=139884119361280 -439> 2018-07-25 11:18:24.196 7f394fbe9700 2 Event(0x56137c9b1040 nevent=5000 time_id=1).set_owner idx=0 owner=139884127753984 -438> 2018-07-25 11:18:24.196 7f395d956280 1 Processor -- start -437> 2018-07-25 11:18:24.196 7f395d956280 1 -- - start start -436> 2018-07-25 11:18:24.196 7f395d956280 10 monclient: init -435> 2018-07-25 11:18:24.196 7f395d956280 5 adding auth protocol: cephx -434> 2018-07-25 11:18:24.196 7f395d956280 10 monclient: auth_supported 2 method cephx -433> 2018-07-25 11:18:24.196 7f395d956280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -432> 2018-07-25 11:18:24.196 7f395d956280 10 monclient: _reopen_session rank -1 -431> 2018-07-25 11:18:24.196 7f395d956280 10 monclient(hunting): picked mon.mimir con 0x56137cdda700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -430> 2018-07-25 11:18:24.196 7f395d956280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56137cd4a780 con 0 -429> 2018-07-25 11:18:24.196 7f395d956280 10 monclient(hunting): _renew_subs -428> 2018-07-25 11:18:24.196 7f395d956280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:23:24.206839 -427> 2018-07-25 11:18:24.196 7f394f3e8700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 -426> 2018-07-25 11:18:24.196 7f394f3e8700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -425> 2018-07-25 11:18:24.196 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 1 0x56137cdec000 mon_map magic: 0 v1 -424> 2018-07-25 11:18:24.196 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 2 0x56137cd4a780 auth_reply(proto 2 0 (0) Success) v1 -423> 2018-07-25 11:18:24.196 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x56137cdec000 con 0x56137cdda700 -422> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -421> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -420> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -419> 2018-07-25 11:18:24.196 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (493333700 0 0) 0x56137cd4a780 con 0x56137cdda700 -418> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient(hunting): my global_id is 220346 -417> 2018-07-25 11:18:24.196 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x56137cd4af00 con 0 -416> 2018-07-25 11:18:24.196 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 3 0x56137cd4af00 auth_reply(proto 2 0 (0) Success) v1 -415> 2018-07-25 11:18:24.196 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3102085623 0 0) 0x56137cd4af00 con 0x56137cdda700 -414> 2018-07-25 11:18:24.196 7f394e3e6700 1 monclient: found mon.mimir -413> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -412> 2018-07-25 11:18:24.196 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56137c9b1680 con 0 -411> 2018-07-25 11:18:24.196 7f394e3e6700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:17:54.213957) -410> 2018-07-25 11:18:24.206 7f394e3e6700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -409> 2018-07-25 11:18:24.206 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x56137cd4a780 con 0 -408> 2018-07-25 11:18:24.206 7f395d956280 5 monclient: authenticate success, global_id 220346 -407> 2018-07-25 11:18:24.206 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 4 0x56137c9b1680 config(0 keys) v1 -406> 2018-07-25 11:18:24.206 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 5 0x56137cdec480 mon_map magic: 0 v1 -405> 2018-07-25 11:18:24.206 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62450 cs=1 l=1). rx mon.2 seq 6 0x56137cd4a780 auth_reply(proto 2 0 (0) Success) v1 -404> 2018-07-25 11:18:24.206 7f394e3e6700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x56137c9b1680 con 0x56137cdda700 -403> 2018-07-25 11:18:24.206 7f394e3e6700 10 monclient: handle_config config(0 keys) v1 -402> 2018-07-25 11:18:24.206 7f394cbe3700 4 set_mon_vals no callback set -401> 2018-07-25 11:18:24.206 7f395d956280 10 monclient: get_monmap_and_config success -400> 2018-07-25 11:18:24.206 7f395d956280 10 monclient: shutdown -399> 2018-07-25 11:18:24.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN pgs=62450 cs=1 l=1).mark_down -398> 2018-07-25 11:18:24.206 7f395d956280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cdda700 :-1 s=STATE_OPEN pgs=62450 cs=1 l=1)._stop -397> 2018-07-25 11:18:24.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 shutdown_connections -396> 2018-07-25 11:18:24.206 7f395d956280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x56137cdda700 -395> 2018-07-25 11:18:24.206 7f395d956280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 shutdown_connections delete 0x56137cdda700 -394> 2018-07-25 11:18:24.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 shutdown_connections -393> 2018-07-25 11:18:24.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 wait complete. -392> 2018-07-25 11:18:24.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 conn(0x56137cdda000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -391> 2018-07-25 11:18:24.206 7f395d956280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/4160920498 conn(0x56137cdda000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -390> 2018-07-25 11:18:24.216 7f395d956280 0 set uid:gid to 114:993 (ceph:ceph) -389> 2018-07-25 11:18:24.216 7f395d956280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1190 -388> 2018-07-25 11:18:24.216 7f395d956280 0 pidfile_write: ignore empty --pid-file -387> 2018-07-25 11:18:24.216 7f395d956280 5 asok(0x56137c9b0240) init /var/run/ceph/ceph-osd.7.asok -386> 2018-07-25 11:18:24.216 7f395d956280 5 asok(0x56137c9b0240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -385> 2018-07-25 11:18:24.216 7f395d956280 -1 asok(0x56137c9b0240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -384> 2018-07-25 11:18:24.216 7f395d956280 5 object store type is bluestore -383> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -382> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 -381> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -380> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1190 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1190 -379> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -378> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1190 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1190 -377> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -376> 2018-07-25 11:18:24.236 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1190 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1190 -375> 2018-07-25 11:18:24.236 7f395d956280 10 monclient: build_initial_monmap -374> 2018-07-25 11:18:24.256 7f395d956280 0 load: jerasure load: lrc load: isa -373> 2018-07-25 11:18:24.256 7f395d956280 5 adding auth protocol: cephx -372> 2018-07-25 11:18:24.256 7f395d956280 5 adding auth protocol: cephx -371> 2018-07-25 11:18:24.256 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -370> 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -369> 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -368> 2018-07-25 11:18:24.256 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -367> 2018-07-25 11:18:24.256 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) close -366> 2018-07-25 11:18:24.586 7f395d956280 5 asok(0x56137c9b0240) register_command objecter_requests hook 0x56137c97a650 -365> 2018-07-25 11:18:24.586 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 start start -364> 2018-07-25 11:18:24.586 7f395d956280 1 -- - start start -363> 2018-07-25 11:18:24.586 7f395d956280 1 -- - start start -362> 2018-07-25 11:18:24.586 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1190 start start -361> 2018-07-25 11:18:24.586 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1190 start start -360> 2018-07-25 11:18:24.586 7f395d956280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1190 start start -359> 2018-07-25 11:18:24.586 7f395d956280 1 -- - start start -358> 2018-07-25 11:18:24.586 7f395d956280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -357> 2018-07-25 11:18:24.586 7f395d956280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -356> 2018-07-25 11:18:24.586 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -355> 2018-07-25 11:18:24.586 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -354> 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -353> 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -352> 2018-07-25 11:18:24.586 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -351> 2018-07-25 11:18:24.586 7f395d956280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -350> 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -349> 2018-07-25 11:18:24.586 7f395d956280 1 bdev(0x56137ceae380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -348> 2018-07-25 11:18:24.586 7f395d956280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -347> 2018-07-25 11:18:24.586 7f395d956280 1 bluefs mount -346> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compaction_readahead_size = 2097152 -345> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compression = kNoCompression -344> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option max_write_buffer_number = 4 -343> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -342> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option recycle_log_file_num = 4 -341> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option writable_file_max_buffer_size = 0 -340> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option write_buffer_size = 268435456 -339> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compaction_readahead_size = 2097152 -338> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option compression = kNoCompression -337> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option max_write_buffer_number = 4 -336> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -335> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option recycle_log_file_num = 4 -334> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option writable_file_max_buffer_size = 0 -333> 2018-07-25 11:18:24.616 7f395d956280 0 set rocksdb option write_buffer_size = 268435456 -332> 2018-07-25 11:18:24.626 7f395d956280 1 rocksdb: do_open column families: [default] -331> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: RocksDB version: 5.13.0 -330> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -329> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Compile date Jul 24 2018 -328> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: DB SUMMARY -327> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: CURRENT file: CURRENT -326> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: IDENTITY file: IDENTITY -325> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: MANIFEST file: MANIFEST-000012 size: 209 Bytes -324> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: SST files in db dir, Total Num: 2, files: 000004.sst 000011.sst -323> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Write Ahead Log file in db: 000013.log size: 4582 ; -322> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.error_if_exists: 0 -321> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.create_if_missing: 0 -320> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.paranoid_checks: 1 -319> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.env: 0x56137ccd53e0 -318> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.info_log: 0x56137ccf7600 -317> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_file_opening_threads: 16 -316> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.statistics: (nil) -315> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_fsync: 0 -314> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_log_file_size: 0 -313> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -312> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.log_file_time_to_roll: 0 -311> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.keep_log_file_num: 1000 -310> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.recycle_log_file_num: 4 -309> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_fallocate: 1 -308> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_mmap_reads: 0 -307> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_mmap_writes: 0 -306> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_direct_reads: 0 -305> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -304> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.create_missing_column_families: 0 -303> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.db_log_dir: -302> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_dir: db -301> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.table_cache_numshardbits: 6 -300> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_subcompactions: 1 -299> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_flushes: -1 -298> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.WAL_ttl_seconds: 0 -297> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.WAL_size_limit_MB: 0 -296> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -295> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.is_fd_close_on_exec: 1 -294> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.advise_random_on_open: 1 -293> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.db_write_buffer_size: 0 -292> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_buffer_manager: 0x56137ccf3c20 -291> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -290> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -289> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -288> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.use_adaptive_mutex: 0 -287> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.rate_limiter: (nil) -286> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -285> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_recovery_mode: 0 -284> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_thread_tracking: 0 -283> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_pipelined_write: 0 -282> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -281> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -280> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -279> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -278> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.row_cache: None -277> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_filter: None -276> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -275> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.allow_ingest_behind: 0 -274> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.preserve_deletes: 0 -273> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.two_write_queues: 0 -272> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.manual_wal_flush: 0 -271> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_jobs: 2 -270> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_background_compactions: -1 -269> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -268> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -267> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.delayed_write_rate : 16777216 -266> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_total_wal_size: 0 -265> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -264> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.stats_dump_period_sec: 600 -263> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.max_open_files: -1 -262> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.bytes_per_sync: 0 -261> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.wal_bytes_per_sync: 0 -260> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Options.compaction_readahead_size: 2097152 -259> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Compression algorithms supported: -258> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -257> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZSTD supported: 0 -256> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kXpressCompression supported: 0 -255> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kLZ4HCCompression supported: 1 -254> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kLZ4Compression supported: 1 -253> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kBZip2Compression supported: 0 -252> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kZlibCompression supported: 1 -251> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: kSnappyCompression supported: 1 -250> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -249> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000012 -248> 2018-07-25 11:18:24.626 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -247> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -246> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -245> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_filter: None -244> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_filter_factory: None -243> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_factory: SkipListFactory -242> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.table_factory: BlockBasedTable -241> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56137c9793c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56137ca2b490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -240> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.write_buffer_size: 268435456 -239> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_write_buffer_number: 4 -238> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression: NoCompression -237> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.bottommost_compression: Disabled -236> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.prefix_extractor: nullptr -235> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -234> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.num_levels: 7 -233> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -232> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -231> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.window_bits: -14 -230> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.level: -1 -229> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.strategy: 0 -228> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -227> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -226> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -225> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -224> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -223> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.target_file_size_base: 67108864 -222> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.target_file_size_multiplier: 1 -221> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -220> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -219> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -218> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -217> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -216> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -215> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -214> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -213> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -212> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -211> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -210> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -209> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.arena_block_size: 33554432 -208> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -207> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -206> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -205> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.disable_auto_compactions: 0 -204> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -203> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -202> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -201> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -200> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -199> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -198> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -197> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -196> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -195> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -194> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -193> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.table_properties_collectors: -192> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.inplace_update_support: 0 -191> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.inplace_update_num_locks: 10000 -190> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -189> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.memtable_huge_page_size: 0 -188> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.bloom_locality: 0 -187> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.max_successive_merges: 0 -186> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.optimize_filters_for_hits: 0 -185> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.paranoid_file_checks: 0 -184> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.force_consistency_checks: 0 -183> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.report_bg_io_stats: 0 -182> 2018-07-25 11:18:24.636 7f395d956280 4 rocksdb: Options.ttl: 0 -181> 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000012 succeeded,manifest_file_number is 12, next_file_number is 15, last_sequence is 20, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 9 -180> 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 10 -179> 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504692284, "job": 1, "event": "recovery_started", "log_files": [13]} -178> 2018-07-25 11:18:24.676 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #13 mode 0 -177> 2018-07-25 11:18:24.686 7f395d956280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:919] [default] [WriteLevel0TableForRecovery] Level-0 table #15: started -176> 2018-07-25 11:18:24.756 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504767930, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 15, "file_size": 5670, "table_properties": {"data_size": 4551, "index_size": 48, "filter_size": 40, "raw_key_size": 191, "raw_average_key_size": 31, "raw_value_size": 4328, "raw_average_value_size": 721, "num_data_blocks": 2, "num_entries": 6, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "1"}} -175> 2018-07-25 11:18:24.756 7f395d956280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:959] [default] [WriteLevel0TableForRecovery] Level-0 table #15: 5670 bytes OK -174> 2018-07-25 11:18:24.756 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 16 -173> 2018-07-25 11:18:24.886 7f395d956280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535504901105, "job": 1, "event": "recovery_finished"} -172> 2018-07-25 11:18:24.886 7f395d956280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000012 type=3 #12 -- OK -171> 2018-07-25 11:18:24.946 7f395d956280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000013.log type=0 #13 -- OK -170> 2018-07-25 11:18:25.026 7f395d956280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x56137cdfd000 -169> 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -168> 2018-07-25 11:18:25.026 7f395d956280 1 freelist init -167> 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -166> 2018-07-25 11:18:25.026 7f395d956280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -165> 2018-07-25 11:18:25.066 7f395d956280 2 osd.7 0 journal looks like hdd -164> 2018-07-25 11:18:25.066 7f395d956280 2 osd.7 0 boot -163> 2018-07-25 11:18:25.066 7f395d956280 0 _get_class not permitted to load kvs -162> 2018-07-25 11:18:25.066 7f395d956280 0 _get_class not permitted to load lua -161> 2018-07-25 11:18:25.066 7f395d956280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -160> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -159> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -158> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -157> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -156> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -155> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -154> 2018-07-25 11:18:25.076 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -153> 2018-07-25 11:18:25.086 7f395d956280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -152> 2018-07-25 11:18:25.096 7f395d956280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -151> 2018-07-25 11:18:25.096 7f395d956280 0 _get_class not permitted to load sdk -150> 2018-07-25 11:18:25.096 7f395d956280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -149> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -148> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -147> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -146> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 load_pgs -145> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 load_pgs opened 0 pgs -144> 2018-07-25 11:18:25.096 7f395d956280 2 osd.7 0 superblock: I am osd.7 -143> 2018-07-25 11:18:25.096 7f395d956280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -142> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -141> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -140> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -139> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -138> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -137> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -136> 2018-07-25 11:18:25.096 7f395d956280 1 Processor -- start -135> 2018-07-25 11:18:25.096 7f395d956280 10 monclient: init -134> 2018-07-25 11:18:25.096 7f395d956280 5 adding auth protocol: cephx -133> 2018-07-25 11:18:25.096 7f395d956280 10 monclient: auth_supported 2 method cephx -132> 2018-07-25 11:18:25.096 7f395d956280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -131> 2018-07-25 11:18:25.096 7f395d956280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -130> 2018-07-25 11:18:25.096 7f395d956280 -1 osd.7 0 log_to_monitors {default=true} -129> 2018-07-25 11:18:25.096 7f395d956280 10 monclient: _reopen_session rank -1 -128> 2018-07-25 11:18:25.106 7f395d956280 10 monclient(hunting): picked mon.mimir con 0x56137cddd800 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -127> 2018-07-25 11:18:25.106 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56137f01a500 con 0 -126> 2018-07-25 11:18:25.106 7f395d956280 10 monclient(hunting): _renew_subs -125> 2018-07-25 11:18:25.106 7f39328eb700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -124> 2018-07-25 11:18:25.106 7f394f3e8700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -123> 2018-07-25 11:18:25.106 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 1 0x56137cdec900 mon_map magic: 0 v1 -122> 2018-07-25 11:18:25.106 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 2 0x56137f01a500 auth_reply(proto 2 0 (0) Success) v1 -121> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x56137cdec900 con 0x56137cddd800 -120> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -119> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -118> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -117> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (4077142302 0 0) 0x56137f01a500 con 0x56137cddd800 -116> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient(hunting): my global_id is 220349 -115> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x56137f01af00 con 0 -114> 2018-07-25 11:18:25.106 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 3 0x56137f01af00 auth_reply(proto 2 0 (0) Success) v1 -113> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (4236098268 0 0) 0x56137f01af00 con 0x56137cddd800 -112> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x56137f01a500 con 0 -111> 2018-07-25 11:18:25.106 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 4 0x56137f01a500 auth_reply(proto 2 0 (0) Success) v1 -110> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (2322736129 0 0) 0x56137f01a500 con 0x56137cddd800 -109> 2018-07-25 11:18:25.106 7f3943d59700 1 monclient: found mon.mimir -108> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -107> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56137cded680 con 0 -106> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:17:55.123501) -105> 2018-07-25 11:18:25.106 7f3943d59700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -104> 2018-07-25 11:18:25.106 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x56137f01af00 con 0 -103> 2018-07-25 11:18:25.106 7f395d956280 5 monclient: authenticate success, global_id 220349 -102> 2018-07-25 11:18:25.106 7f395d956280 10 monclient: wait_auth_rotating waiting (until 2018-07-25 11:18:55.123949) -101> 2018-07-25 11:18:25.116 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 5 0x56137cded680 config(0 keys) v1 -100> 2018-07-25 11:18:25.116 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 6 0x56137f050fc0 mon_map magic: 0 v1 -99> 2018-07-25 11:18:25.116 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 7 0x56137f01af00 auth_reply(proto 2 0 (0) Success) v1 -98> 2018-07-25 11:18:25.116 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x56137cded680 con 0x56137cddd800 -97> 2018-07-25 11:18:25.116 7f3943d59700 10 monclient: handle_config config(0 keys) v1 -96> 2018-07-25 11:18:25.116 7f393a0fa700 4 set_mon_vals no callback set -95> 2018-07-25 11:18:25.116 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x56137f050fc0 con 0x56137cddd800 -94> 2018-07-25 11:18:25.116 7f3943d59700 10 monclient: handle_monmap mon_map magic: 0 v1 -93> 2018-07-25 11:18:25.116 7f3943d59700 10 monclient: got monmap 3, mon.mimir is now rank 2 -92> 2018-07-25 11:18:25.116 7f3943d59700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -91> 2018-07-25 11:18:25.116 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1834935054 0 0) 0x56137f01af00 con 0x56137cddd800 -90> 2018-07-25 11:18:25.116 7f3943d59700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:17:55.129515) -89> 2018-07-25 11:18:25.116 7f395d956280 10 monclient: wait_auth_rotating done -88> 2018-07-25 11:18:25.116 7f395d956280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -87> 2018-07-25 11:18:25.116 7f395d956280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -86> 2018-07-25 11:18:25.116 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x56137cded8c0 con 0 -85> 2018-07-25 11:18:25.206 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 8 0x56137cded8c0 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 -84> 2018-07-25 11:18:25.206 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 ==== 207+0+0 (1704255731 0 0) 0x56137cded8c0 con 0x56137cddd800 -83> 2018-07-25 11:18:25.206 7f3943d59700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -82> 2018-07-25 11:18:25.206 7f3943d59700 10 monclient: _finish_command 1 = 0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. -81> 2018-07-25 11:18:25.206 7f395d956280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -80> 2018-07-25 11:18:25.206 7f395d956280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -79> 2018-07-25 11:18:25.206 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x56137cdedb00 con 0 -78> 2018-07-25 11:18:25.216 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 9 0x56137cdedb00 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 -77> 2018-07-25 11:18:25.216 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 ==== 245+0+0 (3636428954 0 0) 0x56137cdedb00 con 0x56137cddd800 -76> 2018-07-25 11:18:25.216 7f3943d59700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -75> 2018-07-25 11:18:25.216 7f3943d59700 10 monclient: _finish_command 2 = 0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -74> 2018-07-25 11:18:25.216 7f395d956280 0 osd.7 0 done with init, starting boot process -73> 2018-07-25 11:18:25.216 7f395d956280 10 monclient: _renew_subs -72> 2018-07-25 11:18:25.216 7f395d956280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -71> 2018-07-25 11:18:25.216 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x56137c9b18c0 con 0 -70> 2018-07-25 11:18:25.216 7f395d956280 1 osd.7 0 start_boot -69> 2018-07-25 11:18:25.216 7f395d956280 10 monclient: get_version osdmap req 0x56137f046fa0 -68> 2018-07-25 11:18:25.216 7f395d956280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -67> 2018-07-25 11:18:25.216 7f395d956280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x56137ced5800 con 0 -66> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command status hook 0x56137c97a8f0 -65> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command flush_journal hook 0x56137c97a8f0 -64> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_ops_in_flight hook 0x56137c97a8f0 -63> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command ops hook 0x56137c97a8f0 -62> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_blocked_ops hook 0x56137c97a8f0 -61> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_historic_ops hook 0x56137c97a8f0 -60> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_historic_slow_ops hook 0x56137c97a8f0 -59> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_historic_ops_by_duration hook 0x56137c97a8f0 -58> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_op_pq_state hook 0x56137c97a8f0 -57> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_blacklist hook 0x56137c97a8f0 -56> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_watchers hook 0x56137c97a8f0 -55> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_reservations hook 0x56137c97a8f0 -54> 2018-07-25 11:18:25.216 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 10 0x56137cef9800 mgrmap(e 34) v1 -53> 2018-07-25 11:18:25.216 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 11 0x56137cdea000 mon_get_version_reply(handle=1 version=2922) v2 -52> 2018-07-25 11:18:25.216 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x56137cef9800 con 0x56137cddd800 -51> 2018-07-25 11:18:25.216 7f3943d59700 4 mgrc handle_mgr_map Got map version 34 -50> 2018-07-25 11:18:25.216 7f3943d59700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -49> 2018-07-25 11:18:25.216 7f3943d59700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -48> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command get_latest_osdmap hook 0x56137c97a8f0 -47> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command heap hook 0x56137c97a8f0 -46> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command set_heap_property hook 0x56137c97a8f0 -45> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command get_heap_property hook 0x56137c97a8f0 -44> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_objectstore_kv_stats hook 0x56137c97a8f0 -43> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_scrubs hook 0x56137c97a8f0 -42> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command calc_objectstore_db_histogram hook 0x56137c97a8f0 -41> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command flush_store_cache hook 0x56137c97a8f0 -40> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command dump_pgstate_history hook 0x56137c97a8f0 -39> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command compact hook 0x56137c97a8f0 -38> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command get_mapped_pools hook 0x56137c97a8f0 -37> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command smart hook 0x56137c97a8f0 -36> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command list_devices hook 0x56137c97a8f0 -35> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command setomapval hook 0x56137f047320 -34> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command rmomapkey hook 0x56137f047320 -33> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command setomapheader hook 0x56137f047320 -32> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command getomap hook 0x56137f047320 -31> 2018-07-25 11:18:25.216 7f394ebe7700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -30> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command truncobj hook 0x56137f047320 -29> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command injectdataerr hook 0x56137f047320 -28> 2018-07-25 11:18:25.216 7f395d956280 5 asok(0x56137c9b0240) register_command injectmdataerr hook 0x56137f047320 -27> 2018-07-25 11:18:25.226 7f395d956280 5 asok(0x56137c9b0240) register_command set_recovery_delay hook 0x56137f047320 -26> 2018-07-25 11:18:25.226 7f395d956280 5 asok(0x56137c9b0240) register_command trigger_scrub hook 0x56137f047320 -25> 2018-07-25 11:18:25.226 7f395d956280 5 asok(0x56137c9b0240) register_command injectfull hook 0x56137f047320 -24> 2018-07-25 11:18:25.226 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x56137d01e000 con 0 -23> 2018-07-25 11:18:25.226 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2922) v2 ==== 24+0+0 (2719405947 0 0) 0x56137cdea000 con 0x56137cddd800 -22> 2018-07-25 11:18:25.226 7f3943d59700 10 monclient: handle_get_version_reply finishing 0x56137f046fa0 version 2922 -21> 2018-07-25 11:18:25.226 7f393a0fa700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -20> 2018-07-25 11:18:25.226 7f393a0fa700 -1 osd.7 0 waiting for initial osdmap -19> 2018-07-25 11:18:25.226 7f394ebe7700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339630 cs=1 l=1). rx mgr.34109 seq 1 0x56137cf07680 mgrconfigure(period=5, threshold=5) v2 -18> 2018-07-25 11:18:25.226 7f393a0fa700 10 monclient: _renew_subs -17> 2018-07-25 11:18:25.226 7f393a0fa700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -16> 2018-07-25 11:18:25.226 7f393a0fa700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x56137f0518c0 con 0 -15> 2018-07-25 11:18:25.226 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mgr.34109 [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x56137cf07680 con 0x56137d016000 -14> 2018-07-25 11:18:25.226 7f3943d59700 4 mgrc handle_mgr_configure stats_period=5 -13> 2018-07-25 11:18:25.226 7f3943d59700 4 mgrc handle_mgr_configure updated stats threshold: 5 -12> 2018-07-25 11:18:25.226 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgrreport(unknown.7 +54-0 packed 742) v6 -- 0x56137d01e380 con 0 -11> 2018-07-25 11:18:25.236 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- pg_stats(0 pgs tid 0 v 0) v1 -- 0x56137f05a2c0 con 0 -10> 2018-07-25 11:18:25.236 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 12 0x56137f05a580 osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 -9> 2018-07-25 11:18:25.236 7f3943d59700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 ==== 4888+0+0 (2709902744 0 0) 0x56137f05a580 con 0x56137cddd800 -8> 2018-07-25 11:18:25.236 7f3943d59700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2922] -7> 2018-07-25 11:18:25.236 7f394ebe7700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN pgs=339630 cs=1 l=1).read_bulk peer close file descriptor 34 -6> 2018-07-25 11:18:25.236 7f394ebe7700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN pgs=339630 cs=1 l=1).read_until read failed -5> 2018-07-25 11:18:25.236 7f394ebe7700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN pgs=339630 cs=1 l=1).process read tag failed -4> 2018-07-25 11:18:25.236 7f394ebe7700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN pgs=339630 cs=1 l=1).fault on lossy channel, failing -3> 2018-07-25 11:18:25.236 7f394ebe7700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x56137d016000 :-1 s=STATE_OPEN pgs=339630 cs=1 l=1)._stop -2> 2018-07-25 11:18:25.236 7f3943d59700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes -1> 2018-07-25 11:18:25.236 7f394f3e8700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1190 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x56137cddd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62451 cs=1 l=1). rx mon.2 seq 13 0x56137f05a840 osd_map(2320..2359 src has 2319..2922) v4 0> 2018-07-25 11:18:25.246 7f3943d59700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3943d59700 time 2018-07-25 11:18:25.246891 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3954dbeca2] 2: (()+0x283e67) [0x7f3954dbee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x56137a11fefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x56137a0d3ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x56137a0dcd41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x56137a0dd066] 7: (DispatchQueue::entry()+0xb92) [0x7f3954e374a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3954ed4dad] 9: (()+0x792a) [0x7f395330d92a] 10: (clone()+0x3f) [0x7f395229887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:18:25.286 7f3943d59700 -1 *** Caught signal (Aborted) ** in thread 7f3943d59700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x56137a5ef120] 2: (()+0x14120) [0x7f395331a120] 3: (gsignal()+0x110) [0x7f39521cbfc0] 4: (abort()+0x1c7) [0x7f39521cdb07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3954dbedf0] 6: (()+0x283e67) [0x7f3954dbee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x56137a11fefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x56137a0d3ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x56137a0dcd41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x56137a0dd066] 11: (DispatchQueue::entry()+0xb92) [0x7f3954e374a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3954ed4dad] 13: (()+0x792a) [0x7f395330d92a] 14: (clone()+0x3f) [0x7f395229887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:18:25.286 7f3943d59700 -1 *** Caught signal (Aborted) ** in thread 7f3943d59700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x56137a5ef120] 2: (()+0x14120) [0x7f395331a120] 3: (gsignal()+0x110) [0x7f39521cbfc0] 4: (abort()+0x1c7) [0x7f39521cdb07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3954dbedf0] 6: (()+0x283e67) [0x7f3954dbee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x56137a11fefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x56137a0d3ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x56137a0dcd41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x56137a0dd066] 11: (DispatchQueue::entry()+0xb92) [0x7f3954e374a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3954ed4dad] 13: (()+0x792a) [0x7f395330d92a] 14: (clone()+0x3f) [0x7f395229887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:18:49.287 7f8be87e9280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:18:49.287 7f8be87e9280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1282 2018-07-25 11:18:49.287 7f8be87e9280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:18:49.287 7f8be87e9280 -1 asok(0x55be235b2240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:18:49.347 7f8be87e9280 0 load: jerasure load: lrc load: isa 2018-07-25 11:18:49.347 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:49.347 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:49.377 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:49.377 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:18:49.377 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:18:49.707 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:49.717 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev(0x55be23ab0380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev(0x55be23ab0380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:18:49.727 7f8be87e9280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:18:49.727 7f8be87e9280 1 bluefs mount 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:18:49.827 7f8be87e9280 1 rocksdb: do_open column families: [default] 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: DB SUMMARY 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: MANIFEST file: MANIFEST-000016 size: 262 Bytes 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: SST files in db dir, Total Num: 3, files: 000004.sst 000011.sst 000015.sst 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Write Ahead Log file in db: 000017.log size: 78 ; 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.env: 0x55be238d6cc0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.info_log: 0x55be238f9660 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_buffer_manager: 0x55be238f5cb0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.row_cache: None 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000016 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be2357b3c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55be2362d490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:18:49.867 7f8be87e9280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:18:49.867 7f8be87e9280 4 rocksdb: Options.ttl: 0 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000016 succeeded,manifest_file_number is 16, next_file_number is 19, last_sequence is 26, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 13 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 14 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529908992, "job": 1, "event": "recovery_started", "log_files": [17]} 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #17 mode 0 2018-07-25 11:18:49.917 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529925217, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 19, "file_size": 1162, "table_properties": {"data_size": 91, "index_size": 27, "filter_size": 18, "raw_key_size": 55, "raw_average_key_size": 18, "raw_value_size": 16, "raw_average_value_size": 5, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1", "kMergeOperands": "0"}} 2018-07-25 11:18:49.917 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 20 2018-07-25 11:18:49.927 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529941813, "job": 1, "event": "recovery_finished"} 2018-07-25 11:18:49.957 7f8bd27a8700 3 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:490] ------- DUMPING STATS ------- 2018-07-25 11:18:49.957 7f8bd27a8700 3 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:491] ** DB Stats ** Uptime(secs): 0.1 total, 0.1 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 9.54 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Sum 4/0 9.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Uptime(secs): 0.1 total, 0.1 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 9.54 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Sum 4/0 9.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 3] Compacting 4@0 files to L1, score 1.00 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 3 Base level 0, inputs: [19(1162B) 15(5670B) 11(1455B) 4(1484B)] 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529968264, "job": 3, "event": "compaction_started", "files_L0": [19, 15, 11, 4], "score": 1, "input_data_size": 9771} 2018-07-25 11:18:49.957 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55be239fb000 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:18:49.957 7f8be87e9280 1 freelist init 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:18:49.977 7f8be87e9280 0 _get_class not permitted to load kvs 2018-07-25 11:18:49.977 7f8be87e9280 0 _get_class not permitted to load lua 2018-07-25 11:18:49.977 7f8be87e9280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #24: 16 keys, 1833 bytes 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530004538, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 24, "file_size": 1833, "table_properties": {"data_size": 724, "index_size": 28, "filter_size": 50, "raw_key_size": 412, "raw_average_key_size": 25, "raw_value_size": 293, "raw_average_value_size": 18, "num_data_blocks": 1, "num_entries": 16, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "1"}} 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 3] Compacted 4@0 files to L1 => 1833 bytes 2018-07-25 11:18:50.027 7f8be87e9280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: (Original Log Time 2018/07/25-11:18:50.054176) [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.00, MB/sec: 0.3 rd, 0.1 wr, level 1, files in(4, 0) out(1) MB in(0.0, 0.0) out(0.0), read-write-amplify(1.2) write-amplify(0.2) OK, records in: 27, records dropped: 8 output_compression: NoCompression 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: (Original Log Time 2018/07/25-11:18:50.054247) EVENT_LOG_v1 {"time_micros": 1532535530054219, "job": 3, "event": "compaction_finished", "compaction_time_micros": 36367, "output_level": 1, "num_output_files": 1, "total_output_size": 1833, "num_input_records": 24, "num_output_records": 16, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054610, "job": 3, "event": "table_file_deletion", "file_number": 19} 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054658, "job": 3, "event": "table_file_deletion", "file_number": 15} 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054704, "job": 3, "event": "table_file_deletion", "file_number": 11} 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054749, "job": 3, "event": "table_file_deletion", "file_number": 4} 2018-07-25 11:18:50.047 7f8be87e9280 0 _get_class not permitted to load sdk 2018-07-25 11:18:50.047 7f8be87e9280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 load_pgs 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:18:50.047 7f8be87e9280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:18:50.147 7f8be87e9280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:18:50.147 7f8be87e9280 1 osd.7 0 start_boot 2018-07-25 11:18:50.157 7f8bc4f8d700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:18:50.157 7f8bcebec700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:18:50.187 7f8bcebec700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f8bcebec700 time 2018-07-25 11:18:50.171269 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f8bdfc51ca2] 2: (()+0x283e67) [0x7f8bdfc51e67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55be20fafefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55be20f63ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55be20f6cd41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55be20f6d066] 7: (DispatchQueue::entry()+0xb92) [0x7f8bdfcca4a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f8bdfd67dad] 9: (()+0x792a) [0x7f8bde1a092a] 10: (clone()+0x3f) [0x7f8bdd12b87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -472> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perfcounters_dump hook 0x55be2357c060 -471> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command 1 hook 0x55be2357c060 -470> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perf dump hook 0x55be2357c060 -469> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perfcounters_schema hook 0x55be2357c060 -468> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perf histogram dump hook 0x55be2357c060 -467> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command 2 hook 0x55be2357c060 -466> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perf schema hook 0x55be2357c060 -465> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perf histogram schema hook 0x55be2357c060 -464> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command perf reset hook 0x55be2357c060 -463> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config show hook 0x55be2357c060 -462> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config help hook 0x55be2357c060 -461> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config set hook 0x55be2357c060 -460> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config unset hook 0x55be2357c060 -459> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config get hook 0x55be2357c060 -458> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config diff hook 0x55be2357c060 -457> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command config diff get hook 0x55be2357c060 -456> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command log flush hook 0x55be2357c060 -455> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command log dump hook 0x55be2357c060 -454> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command log reopen hook 0x55be2357c060 -453> 2018-07-25 11:18:49.127 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_mempools hook 0x55be238fe068 -452> 2018-07-25 11:18:49.157 7f8be87e9280 10 monclient: get_monmap_and_config -451> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient: build_initial_monmap -450> 2018-07-25 11:18:49.187 7f8bdaa7c700 2 Event(0x55be235b3040 nevent=5000 time_id=1).set_owner idx=0 owner=140238645610240 -449> 2018-07-25 11:18:49.187 7f8bd9a7a700 2 Event(0x55be235b34c0 nevent=5000 time_id=1).set_owner idx=2 owner=140238628824832 -448> 2018-07-25 11:18:49.187 7f8bda27b700 2 Event(0x55be235b3280 nevent=5000 time_id=1).set_owner idx=1 owner=140238637217536 -447> 2018-07-25 11:18:49.187 7f8be87e9280 1 Processor -- start -446> 2018-07-25 11:18:49.187 7f8be87e9280 1 -- - start start -445> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient: init -444> 2018-07-25 11:18:49.187 7f8be87e9280 5 adding auth protocol: cephx -443> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient: auth_supported 2 method cephx -442> 2018-07-25 11:18:49.187 7f8be87e9280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -441> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient: _reopen_session rank -1 -440> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient(hunting): picked mon.mimir con 0x55be239dc700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -439> 2018-07-25 11:18:49.187 7f8be87e9280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55be2394c780 con 0 -438> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient(hunting): _renew_subs -437> 2018-07-25 11:18:49.187 7f8be87e9280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:23:49.198709 -436> 2018-07-25 11:18:49.217 7f8bda27b700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 -435> 2018-07-25 11:18:49.227 7f8bda27b700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -434> 2018-07-25 11:18:49.237 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62452 cs=1 l=1). rx mon.2 seq 1 0x55be239ee000 mon_map magic: 0 v1 -433> 2018-07-25 11:18:49.237 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62452 cs=1 l=1). rx mon.2 seq 2 0x55be2394c780 auth_reply(proto 2 0 (0) Success) v1 -432> 2018-07-25 11:18:49.237 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55be239ee000 con 0x55be239dc700 -431> 2018-07-25 11:18:49.237 7f8bd9279700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -430> 2018-07-25 11:18:49.237 7f8bd9279700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -429> 2018-07-25 11:18:49.237 7f8bd9279700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -428> 2018-07-25 11:18:49.237 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3337258225 0 0) 0x55be2394c780 con 0x55be239dc700 -427> 2018-07-25 11:18:49.237 7f8bd9279700 10 monclient(hunting): my global_id is 220352 -426> 2018-07-25 11:18:49.237 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55be2394cf00 con 0 -425> 2018-07-25 11:18:49.257 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62452 cs=1 l=1). rx mon.2 seq 3 0x55be2394cf00 auth_reply(proto 2 0 (0) Success) v1 -424> 2018-07-25 11:18:49.257 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (654551638 0 0) 0x55be2394cf00 con 0x55be239dc700 -423> 2018-07-25 11:18:49.257 7f8bd9279700 1 monclient: found mon.mimir -422> 2018-07-25 11:18:49.257 7f8bd9279700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -421> 2018-07-25 11:18:49.257 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be235b3680 con 0 -420> 2018-07-25 11:18:49.257 7f8bd9279700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:18:19.264638) -419> 2018-07-25 11:18:49.257 7f8bd9279700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -418> 2018-07-25 11:18:49.257 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55be2394c780 con 0 -417> 2018-07-25 11:18:49.257 7f8be87e9280 5 monclient: authenticate success, global_id 220352 -416> 2018-07-25 11:18:49.257 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62452 cs=1 l=1). rx mon.2 seq 4 0x55be235b3680 config(0 keys) v1 -415> 2018-07-25 11:18:49.257 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55be235b3680 con 0x55be239dc700 -414> 2018-07-25 11:18:49.257 7f8bd9279700 10 monclient: handle_config config(0 keys) v1 -413> 2018-07-25 11:18:49.257 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62452 cs=1 l=1). rx mon.2 seq 5 0x55be239ee480 mon_map magic: 0 v1 -412> 2018-07-25 11:18:49.257 7f8be87e9280 10 monclient: get_monmap_and_config success -411> 2018-07-25 11:18:49.257 7f8be87e9280 10 monclient: shutdown -410> 2018-07-25 11:18:49.257 7f8bd9279700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55be239ee480 con 0x55be239dc700 -409> 2018-07-25 11:18:49.257 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN pgs=62452 cs=1 l=1).mark_down -408> 2018-07-25 11:18:49.257 7f8be87e9280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be239dc700 :-1 s=STATE_OPEN pgs=62452 cs=1 l=1)._stop -407> 2018-07-25 11:18:49.257 7f8bd9279700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 -406> 2018-07-25 11:18:49.257 7f8bd7a76700 4 set_mon_vals no callback set -405> 2018-07-25 11:18:49.277 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 shutdown_connections -404> 2018-07-25 11:18:49.277 7f8be87e9280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x55be239dc700 -403> 2018-07-25 11:18:49.277 7f8be87e9280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 shutdown_connections delete 0x55be239dc700 -402> 2018-07-25 11:18:49.277 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 shutdown_connections -401> 2018-07-25 11:18:49.277 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 wait complete. -400> 2018-07-25 11:18:49.277 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 conn(0x55be239dc000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -399> 2018-07-25 11:18:49.277 7f8be87e9280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2281880848 conn(0x55be239dc000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -398> 2018-07-25 11:18:49.287 7f8be87e9280 0 set uid:gid to 114:993 (ceph:ceph) -397> 2018-07-25 11:18:49.287 7f8be87e9280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1282 -396> 2018-07-25 11:18:49.287 7f8be87e9280 0 pidfile_write: ignore empty --pid-file -395> 2018-07-25 11:18:49.287 7f8be87e9280 5 asok(0x55be235b2240) init /var/run/ceph/ceph-osd.7.asok -394> 2018-07-25 11:18:49.287 7f8be87e9280 5 asok(0x55be235b2240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -393> 2018-07-25 11:18:49.287 7f8be87e9280 -1 asok(0x55be235b2240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -392> 2018-07-25 11:18:49.287 7f8be87e9280 5 object store type is bluestore -391> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -390> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 -389> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -388> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1282 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1282 -387> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -386> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1282 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1282 -385> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -384> 2018-07-25 11:18:49.327 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1282 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1282 -383> 2018-07-25 11:18:49.327 7f8be87e9280 10 monclient: build_initial_monmap -382> 2018-07-25 11:18:49.347 7f8be87e9280 0 load: jerasure load: lrc load: isa -381> 2018-07-25 11:18:49.347 7f8be87e9280 5 adding auth protocol: cephx -380> 2018-07-25 11:18:49.347 7f8be87e9280 5 adding auth protocol: cephx -379> 2018-07-25 11:18:49.347 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -378> 2018-07-25 11:18:49.347 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -377> 2018-07-25 11:18:49.377 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -376> 2018-07-25 11:18:49.377 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -375> 2018-07-25 11:18:49.377 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) close -374> 2018-07-25 11:18:49.707 7f8be87e9280 5 asok(0x55be235b2240) register_command objecter_requests hook 0x55be2357c650 -373> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 start start -372> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- - start start -371> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- - start start -370> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1282 start start -369> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1282 start start -368> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1282 start start -367> 2018-07-25 11:18:49.707 7f8be87e9280 1 -- - start start -366> 2018-07-25 11:18:49.707 7f8be87e9280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -365> 2018-07-25 11:18:49.707 7f8be87e9280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -364> 2018-07-25 11:18:49.707 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -363> 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -362> 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -361> 2018-07-25 11:18:49.707 7f8be87e9280 1 bdev(0x55be23ab0000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -360> 2018-07-25 11:18:49.717 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -359> 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -358> 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev(0x55be23ab0380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -357> 2018-07-25 11:18:49.717 7f8be87e9280 1 bdev(0x55be23ab0380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -356> 2018-07-25 11:18:49.727 7f8be87e9280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -355> 2018-07-25 11:18:49.727 7f8be87e9280 1 bluefs mount -354> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compaction_readahead_size = 2097152 -353> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compression = kNoCompression -352> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option max_write_buffer_number = 4 -351> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -350> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option recycle_log_file_num = 4 -349> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option writable_file_max_buffer_size = 0 -348> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option write_buffer_size = 268435456 -347> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compaction_readahead_size = 2097152 -346> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option compression = kNoCompression -345> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option max_write_buffer_number = 4 -344> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -343> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option recycle_log_file_num = 4 -342> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option writable_file_max_buffer_size = 0 -341> 2018-07-25 11:18:49.817 7f8be87e9280 0 set rocksdb option write_buffer_size = 268435456 -340> 2018-07-25 11:18:49.827 7f8be87e9280 1 rocksdb: do_open column families: [default] -339> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: RocksDB version: 5.13.0 -338> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -337> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Compile date Jul 24 2018 -336> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: DB SUMMARY -335> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: CURRENT file: CURRENT -334> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: IDENTITY file: IDENTITY -333> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: MANIFEST file: MANIFEST-000016 size: 262 Bytes -332> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: SST files in db dir, Total Num: 3, files: 000004.sst 000011.sst 000015.sst -331> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Write Ahead Log file in db: 000017.log size: 78 ; -330> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.error_if_exists: 0 -329> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.create_if_missing: 0 -328> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.paranoid_checks: 1 -327> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.env: 0x55be238d6cc0 -326> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.info_log: 0x55be238f9660 -325> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_file_opening_threads: 16 -324> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.statistics: (nil) -323> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_fsync: 0 -322> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_log_file_size: 0 -321> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -320> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.log_file_time_to_roll: 0 -319> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.keep_log_file_num: 1000 -318> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.recycle_log_file_num: 4 -317> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_fallocate: 1 -316> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_mmap_reads: 0 -315> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_mmap_writes: 0 -314> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_direct_reads: 0 -313> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -312> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.create_missing_column_families: 0 -311> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.db_log_dir: -310> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_dir: db -309> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.table_cache_numshardbits: 6 -308> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_subcompactions: 1 -307> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_flushes: -1 -306> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.WAL_ttl_seconds: 0 -305> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.WAL_size_limit_MB: 0 -304> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -303> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.is_fd_close_on_exec: 1 -302> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.advise_random_on_open: 1 -301> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.db_write_buffer_size: 0 -300> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_buffer_manager: 0x55be238f5cb0 -299> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -298> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -297> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -296> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.use_adaptive_mutex: 0 -295> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.rate_limiter: (nil) -294> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -293> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_recovery_mode: 0 -292> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_thread_tracking: 0 -291> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_pipelined_write: 0 -290> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -289> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -288> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -287> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -286> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.row_cache: None -285> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_filter: None -284> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -283> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.allow_ingest_behind: 0 -282> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.preserve_deletes: 0 -281> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.two_write_queues: 0 -280> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.manual_wal_flush: 0 -279> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_jobs: 2 -278> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_background_compactions: -1 -277> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -276> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -275> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.delayed_write_rate : 16777216 -274> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_total_wal_size: 0 -273> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -272> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.stats_dump_period_sec: 600 -271> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.max_open_files: -1 -270> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.bytes_per_sync: 0 -269> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.wal_bytes_per_sync: 0 -268> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_readahead_size: 2097152 -267> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Compression algorithms supported: -266> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -265> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZSTD supported: 0 -264> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kXpressCompression supported: 0 -263> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kLZ4HCCompression supported: 1 -262> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kLZ4Compression supported: 1 -261> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kBZip2Compression supported: 0 -260> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kZlibCompression supported: 1 -259> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: kSnappyCompression supported: 1 -258> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -257> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000016 -256> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -255> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -254> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -253> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_filter: None -252> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.compaction_filter_factory: None -251> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.memtable_factory: SkipListFactory -250> 2018-07-25 11:18:49.837 7f8be87e9280 4 rocksdb: Options.table_factory: BlockBasedTable -249> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be2357b3c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55be2362d490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -248> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.write_buffer_size: 268435456 -247> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_write_buffer_number: 4 -246> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression: NoCompression -245> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.bottommost_compression: Disabled -244> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.prefix_extractor: nullptr -243> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -242> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.num_levels: 7 -241> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -240> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -239> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.window_bits: -14 -238> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.level: -1 -237> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.strategy: 0 -236> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -235> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -234> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -233> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -232> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -231> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.target_file_size_base: 67108864 -230> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.target_file_size_multiplier: 1 -229> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -228> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -227> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -226> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -225> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -224> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -223> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -222> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -221> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -220> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -219> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -218> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -217> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.arena_block_size: 33554432 -216> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -215> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -214> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -213> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.disable_auto_compactions: 0 -212> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -211> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -210> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -209> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -208> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -207> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -206> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -205> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -204> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -203> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -202> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -201> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.table_properties_collectors: -200> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.inplace_update_support: 0 -199> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.inplace_update_num_locks: 10000 -198> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -197> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.memtable_huge_page_size: 0 -196> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.bloom_locality: 0 -195> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.max_successive_merges: 0 -194> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.optimize_filters_for_hits: 0 -193> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.paranoid_file_checks: 0 -192> 2018-07-25 11:18:49.847 7f8be87e9280 4 rocksdb: Options.force_consistency_checks: 0 -191> 2018-07-25 11:18:49.867 7f8be87e9280 4 rocksdb: Options.report_bg_io_stats: 0 -190> 2018-07-25 11:18:49.867 7f8be87e9280 4 rocksdb: Options.ttl: 0 -189> 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000016 succeeded,manifest_file_number is 16, next_file_number is 19, last_sequence is 26, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 13 -188> 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 14 -187> 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529908992, "job": 1, "event": "recovery_started", "log_files": [17]} -186> 2018-07-25 11:18:49.897 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #17 mode 0 -185> 2018-07-25 11:18:49.897 7f8be87e9280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:919] [default] [WriteLevel0TableForRecovery] Level-0 table #19: started -184> 2018-07-25 11:18:49.917 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529925217, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 19, "file_size": 1162, "table_properties": {"data_size": 91, "index_size": 27, "filter_size": 18, "raw_key_size": 55, "raw_average_key_size": 18, "raw_value_size": 16, "raw_average_value_size": 5, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1", "kMergeOperands": "0"}} -183> 2018-07-25 11:18:49.917 7f8be87e9280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:959] [default] [WriteLevel0TableForRecovery] Level-0 table #19: 1162 bytes OK -182> 2018-07-25 11:18:49.917 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 20 -181> 2018-07-25 11:18:49.927 7f8be87e9280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529941813, "job": 1, "event": "recovery_finished"} -180> 2018-07-25 11:18:49.927 7f8be87e9280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000016 type=3 #16 -- OK -179> 2018-07-25 11:18:49.937 7f8be87e9280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000017.log type=0 #17 -- OK -178> 2018-07-25 11:18:49.957 7f8bd27a8700 3 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:490] ------- DUMPING STATS ------- -177> 2018-07-25 11:18:49.957 7f8bd27a8700 3 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl.cc:491] ** DB Stats ** Uptime(secs): 0.1 total, 0.1 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 9.54 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Sum 4/0 9.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Uptime(secs): 0.1 total, 0.1 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 9.54 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Sum 4/0 9.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0 1 0.014 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** -176> 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 3] Compacting 4@0 files to L1, score 1.00 -175> 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 3 Base level 0, inputs: [19(1162B) 15(5670B) 11(1455B) 4(1484B)] -174> 2018-07-25 11:18:49.957 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535529968264, "job": 3, "event": "compaction_started", "files_L0": [19, 15, 11, 4], "score": 1, "input_data_size": 9771} -173> 2018-07-25 11:18:49.957 7f8be87e9280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55be239fb000 -172> 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -171> 2018-07-25 11:18:49.957 7f8be87e9280 1 freelist init -170> 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -169> 2018-07-25 11:18:49.957 7f8be87e9280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -168> 2018-07-25 11:18:49.957 7f8be87e9280 2 osd.7 0 journal looks like hdd -167> 2018-07-25 11:18:49.957 7f8be87e9280 2 osd.7 0 boot -166> 2018-07-25 11:18:49.977 7f8be87e9280 0 _get_class not permitted to load kvs -165> 2018-07-25 11:18:49.977 7f8be87e9280 0 _get_class not permitted to load lua -164> 2018-07-25 11:18:49.977 7f8be87e9280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -163> 2018-07-25 11:18:49.977 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -162> 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #24: 16 keys, 1833 bytes -161> 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530004538, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 24, "file_size": 1833, "table_properties": {"data_size": 724, "index_size": 28, "filter_size": 50, "raw_key_size": 412, "raw_average_key_size": 25, "raw_value_size": 293, "raw_average_value_size": 18, "num_data_blocks": 1, "num_entries": 16, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "1"}} -160> 2018-07-25 11:18:49.997 7f8bd27a8700 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 3] Compacted 4@0 files to L1 => 1833 bytes -159> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -158> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -157> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -156> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -155> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -154> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -153> 2018-07-25 11:18:49.997 7f8be87e9280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -152> 2018-07-25 11:18:50.027 7f8be87e9280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -151> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: (Original Log Time 2018/07/25-11:18:50.054176) [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.00, MB/sec: 0.3 rd, 0.1 wr, level 1, files in(4, 0) out(1) MB in(0.0, 0.0) out(0.0), read-write-amplify(1.2) write-amplify(0.2) OK, records in: 27, records dropped: 8 output_compression: NoCompression -150> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: (Original Log Time 2018/07/25-11:18:50.054247) EVENT_LOG_v1 {"time_micros": 1532535530054219, "job": 3, "event": "compaction_finished", "compaction_time_micros": 36367, "output_level": 1, "num_output_files": 1, "total_output_size": 1833, "num_input_records": 24, "num_output_records": 16, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} -149> 2018-07-25 11:18:50.047 7f8bd27a8700 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 3] Delete db/000019.sst type=2 #19 -- OK -148> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054610, "job": 3, "event": "table_file_deletion", "file_number": 19} -147> 2018-07-25 11:18:50.047 7f8bd27a8700 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 3] Delete db/000015.sst type=2 #15 -- OK -146> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054658, "job": 3, "event": "table_file_deletion", "file_number": 15} -145> 2018-07-25 11:18:50.047 7f8bd27a8700 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 3] Delete db/000011.sst type=2 #11 -- OK -144> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054704, "job": 3, "event": "table_file_deletion", "file_number": 11} -143> 2018-07-25 11:18:50.047 7f8bd27a8700 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 3] Delete db/000004.sst type=2 #4 -- OK -142> 2018-07-25 11:18:50.047 7f8bd27a8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535530054749, "job": 3, "event": "table_file_deletion", "file_number": 4} -141> 2018-07-25 11:18:50.047 7f8be87e9280 0 _get_class not permitted to load sdk -140> 2018-07-25 11:18:50.047 7f8be87e9280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -139> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -138> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -137> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -136> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 load_pgs -135> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 load_pgs opened 0 pgs -134> 2018-07-25 11:18:50.047 7f8be87e9280 2 osd.7 0 superblock: I am osd.7 -133> 2018-07-25 11:18:50.047 7f8be87e9280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -132> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -131> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -130> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -129> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -128> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -127> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -126> 2018-07-25 11:18:50.047 7f8be87e9280 1 Processor -- start -125> 2018-07-25 11:18:50.047 7f8be87e9280 10 monclient: init -124> 2018-07-25 11:18:50.047 7f8be87e9280 5 adding auth protocol: cephx -123> 2018-07-25 11:18:50.047 7f8be87e9280 10 monclient: auth_supported 2 method cephx -122> 2018-07-25 11:18:50.047 7f8be87e9280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -121> 2018-07-25 11:18:50.047 7f8be87e9280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -120> 2018-07-25 11:18:50.047 7f8be87e9280 -1 osd.7 0 log_to_monitors {default=true} -119> 2018-07-25 11:18:50.047 7f8be87e9280 10 monclient: _reopen_session rank -1 -118> 2018-07-25 11:18:50.047 7f8be87e9280 10 monclient(hunting): picked mon.mimir con 0x55be23c1a000 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -117> 2018-07-25 11:18:50.047 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55be23c18500 con 0 -116> 2018-07-25 11:18:50.047 7f8be87e9280 10 monclient(hunting): _renew_subs -115> 2018-07-25 11:18:50.047 7f8bbd77e700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -114> 2018-07-25 11:18:50.067 7f8bda27b700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -113> 2018-07-25 11:18:50.077 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 1 0x55be239ee900 mon_map magic: 0 v1 -112> 2018-07-25 11:18:50.077 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 2 0x55be23c18500 auth_reply(proto 2 0 (0) Success) v1 -111> 2018-07-25 11:18:50.077 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55be239ee900 con 0x55be23c1a000 -110> 2018-07-25 11:18:50.077 7f8bcebec700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -109> 2018-07-25 11:18:50.077 7f8bcebec700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -108> 2018-07-25 11:18:50.077 7f8bcebec700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -107> 2018-07-25 11:18:50.077 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3910026393 0 0) 0x55be23c18500 con 0x55be23c1a000 -106> 2018-07-25 11:18:50.077 7f8bcebec700 10 monclient(hunting): my global_id is 220355 -105> 2018-07-25 11:18:50.077 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55be23c19180 con 0 -104> 2018-07-25 11:18:50.087 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 3 0x55be23c19180 auth_reply(proto 2 0 (0) Success) v1 -103> 2018-07-25 11:18:50.087 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1710726778 0 0) 0x55be23c19180 con 0x55be23c1a000 -102> 2018-07-25 11:18:50.087 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55be23c18500 con 0 -101> 2018-07-25 11:18:50.097 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 4 0x55be23c18500 auth_reply(proto 2 0 (0) Success) v1 -100> 2018-07-25 11:18:50.097 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (3217356400 0 0) 0x55be23c18500 con 0x55be23c1a000 -99> 2018-07-25 11:18:50.097 7f8bcebec700 1 monclient: found mon.mimir -98> 2018-07-25 11:18:50.097 7f8bcebec700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -97> 2018-07-25 11:18:50.097 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be239ef8c0 con 0 -96> 2018-07-25 11:18:50.097 7f8bcebec700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:18:20.104537) -95> 2018-07-25 11:18:50.097 7f8bcebec700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -94> 2018-07-25 11:18:50.097 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55be23c19180 con 0 -93> 2018-07-25 11:18:50.097 7f8be87e9280 5 monclient: authenticate success, global_id 220355 -92> 2018-07-25 11:18:50.097 7f8be87e9280 10 monclient: wait_auth_rotating waiting (until 2018-07-25 11:19:20.104771) -91> 2018-07-25 11:18:50.107 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 5 0x55be239ef8c0 config(0 keys) v1 -90> 2018-07-25 11:18:50.107 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 6 0x55be23c22480 mon_map magic: 0 v1 -89> 2018-07-25 11:18:50.107 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 7 0x55be23c19180 auth_reply(proto 2 0 (0) Success) v1 -88> 2018-07-25 11:18:50.107 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55be239ef8c0 con 0x55be23c1a000 -87> 2018-07-25 11:18:50.107 7f8bcebec700 10 monclient: handle_config config(0 keys) v1 -86> 2018-07-25 11:18:50.107 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55be23c22480 con 0x55be23c1a000 -85> 2018-07-25 11:18:50.107 7f8bcebec700 10 monclient: handle_monmap mon_map magic: 0 v1 -84> 2018-07-25 11:18:50.107 7f8bcebec700 10 monclient: got monmap 3, mon.mimir is now rank 2 -83> 2018-07-25 11:18:50.107 7f8bcebec700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -82> 2018-07-25 11:18:50.107 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3158028839 0 0) 0x55be23c19180 con 0x55be23c1a000 -81> 2018-07-25 11:18:50.107 7f8bcebec700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:18:20.114653) -80> 2018-07-25 11:18:50.107 7f8be87e9280 10 monclient: wait_auth_rotating done -79> 2018-07-25 11:18:50.107 7f8be87e9280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -78> 2018-07-25 11:18:50.107 7f8be87e9280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -77> 2018-07-25 11:18:50.107 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x55be23c22b40 con 0 -76> 2018-07-25 11:18:50.117 7f8bc4f8d700 4 set_mon_vals no callback set -75> 2018-07-25 11:18:50.117 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 8 0x55be23c22b40 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 -74> 2018-07-25 11:18:50.117 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 ==== 207+0+0 (1704255731 0 0) 0x55be23c22b40 con 0x55be23c1a000 -73> 2018-07-25 11:18:50.117 7f8bcebec700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -72> 2018-07-25 11:18:50.117 7f8bcebec700 10 monclient: _finish_command 1 = 0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. -71> 2018-07-25 11:18:50.127 7f8be87e9280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -70> 2018-07-25 11:18:50.127 7f8be87e9280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -69> 2018-07-25 11:18:50.127 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x55be23c22d80 con 0 -68> 2018-07-25 11:18:50.147 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 9 0x55be23c22d80 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 -67> 2018-07-25 11:18:50.147 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 ==== 245+0+0 (3636428954 0 0) 0x55be23c22d80 con 0x55be23c1a000 -66> 2018-07-25 11:18:50.147 7f8bcebec700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -65> 2018-07-25 11:18:50.147 7f8bcebec700 10 monclient: _finish_command 2 = 0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -64> 2018-07-25 11:18:50.147 7f8be87e9280 0 osd.7 0 done with init, starting boot process -63> 2018-07-25 11:18:50.147 7f8be87e9280 10 monclient: _renew_subs -62> 2018-07-25 11:18:50.147 7f8be87e9280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -61> 2018-07-25 11:18:50.147 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x55be239ee240 con 0 -60> 2018-07-25 11:18:50.147 7f8be87e9280 1 osd.7 0 start_boot -59> 2018-07-25 11:18:50.147 7f8be87e9280 10 monclient: get_version osdmap req 0x55be23b0e760 -58> 2018-07-25 11:18:50.147 7f8be87e9280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -57> 2018-07-25 11:18:50.147 7f8be87e9280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x55be23ad9800 con 0 -56> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command status hook 0x55be2357c8e0 -55> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command flush_journal hook 0x55be2357c8e0 -54> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_ops_in_flight hook 0x55be2357c8e0 -53> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command ops hook 0x55be2357c8e0 -52> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_blocked_ops hook 0x55be2357c8e0 -51> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_historic_ops hook 0x55be2357c8e0 -50> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_historic_slow_ops hook 0x55be2357c8e0 -49> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_historic_ops_by_duration hook 0x55be2357c8e0 -48> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_op_pq_state hook 0x55be2357c8e0 -47> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_blacklist hook 0x55be2357c8e0 -46> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_watchers hook 0x55be2357c8e0 -45> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_reservations hook 0x55be2357c8e0 -44> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command get_latest_osdmap hook 0x55be2357c8e0 -43> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command heap hook 0x55be2357c8e0 -42> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command set_heap_property hook 0x55be2357c8e0 -41> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command get_heap_property hook 0x55be2357c8e0 -40> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_objectstore_kv_stats hook 0x55be2357c8e0 -39> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_scrubs hook 0x55be2357c8e0 -38> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command calc_objectstore_db_histogram hook 0x55be2357c8e0 -37> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command flush_store_cache hook 0x55be2357c8e0 -36> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command dump_pgstate_history hook 0x55be2357c8e0 -35> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command compact hook 0x55be2357c8e0 -34> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command get_mapped_pools hook 0x55be2357c8e0 -33> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command smart hook 0x55be2357c8e0 -32> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command list_devices hook 0x55be2357c8e0 -31> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command setomapval hook 0x55be23b0ea00 -30> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command rmomapkey hook 0x55be23b0ea00 -29> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command setomapheader hook 0x55be23b0ea00 -28> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command getomap hook 0x55be23b0ea00 -27> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command truncobj hook 0x55be23b0ea00 -26> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command injectdataerr hook 0x55be23b0ea00 -25> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command injectmdataerr hook 0x55be23b0ea00 -24> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command set_recovery_delay hook 0x55be23b0ea00 -23> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command trigger_scrub hook 0x55be23b0ea00 -22> 2018-07-25 11:18:50.147 7f8be87e9280 5 asok(0x55be235b2240) register_command injectfull hook 0x55be23b0ea00 -21> 2018-07-25 11:18:50.157 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 10 0x55be23afb800 mgrmap(e 34) v1 -20> 2018-07-25 11:18:50.157 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 11 0x55be239ec000 mon_get_version_reply(handle=1 version=2922) v2 -19> 2018-07-25 11:18:50.157 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x55be23afb800 con 0x55be23c1a000 -18> 2018-07-25 11:18:50.157 7f8bcebec700 4 mgrc handle_mgr_map Got map version 34 -17> 2018-07-25 11:18:50.157 7f8bcebec700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -16> 2018-07-25 11:18:50.157 7f8bcebec700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -15> 2018-07-25 11:18:50.157 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x55be23f3a380 con 0 -14> 2018-07-25 11:18:50.157 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2922) v2 ==== 24+0+0 (2719405947 0 0) 0x55be239ec000 con 0x55be23c1a000 -13> 2018-07-25 11:18:50.157 7f8bcebec700 10 monclient: handle_get_version_reply finishing 0x55be23b0e760 version 2922 -12> 2018-07-25 11:18:50.157 7f8bc4f8d700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -11> 2018-07-25 11:18:50.157 7f8bc4f8d700 -1 osd.7 0 waiting for initial osdmap -10> 2018-07-25 11:18:50.157 7f8bc4f8d700 10 monclient: _renew_subs -9> 2018-07-25 11:18:50.157 7f8bc4f8d700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -8> 2018-07-25 11:18:50.157 7f8bc4f8d700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x55be239ef8c0 con 0 -7> 2018-07-25 11:18:50.157 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 12 0x55be24551080 osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 -6> 2018-07-25 11:18:50.157 7f8bcebec700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 ==== 4888+0+0 (2709902744 0 0) 0x55be24551080 con 0x55be23c1a000 -5> 2018-07-25 11:18:50.157 7f8bcebec700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2922] -4> 2018-07-25 11:18:50.157 7f8bcebec700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes -3> 2018-07-25 11:18:50.157 7f8bd9a7a700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55be23c1ca00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -2> 2018-07-25 11:18:50.167 7f8bda27b700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55be23c1a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62453 cs=1 l=1). rx mon.2 seq 13 0x55be24551340 osd_map(2320..2359 src has 2319..2922) v4 -1> 2018-07-25 11:18:50.167 7f8bd9a7a700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1282 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55be23c1ca00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339631 cs=1 l=1). rx mgr.34109 seq 1 0x55be23b11860 mgrconfigure(period=5, threshold=5) v2 0> 2018-07-25 11:18:50.187 7f8bcebec700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f8bcebec700 time 2018-07-25 11:18:50.171269 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f8bdfc51ca2] 2: (()+0x283e67) [0x7f8bdfc51e67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55be20fafefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55be20f63ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55be20f6cd41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55be20f6d066] 7: (DispatchQueue::entry()+0xb92) [0x7f8bdfcca4a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f8bdfd67dad] 9: (()+0x792a) [0x7f8bde1a092a] 10: (clone()+0x3f) [0x7f8bdd12b87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:18:50.247 7f8bcebec700 -1 *** Caught signal (Aborted) ** in thread 7f8bcebec700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55be2147f120] 2: (()+0x14120) [0x7f8bde1ad120] 3: (gsignal()+0x110) [0x7f8bdd05efc0] 4: (abort()+0x1c7) [0x7f8bdd060b07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f8bdfc51df0] 6: (()+0x283e67) [0x7f8bdfc51e67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55be20fafefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55be20f63ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55be20f6cd41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55be20f6d066] 11: (DispatchQueue::entry()+0xb92) [0x7f8bdfcca4a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f8bdfd67dad] 13: (()+0x792a) [0x7f8bde1a092a] 14: (clone()+0x3f) [0x7f8bdd12b87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:18:50.247 7f8bcebec700 -1 *** Caught signal (Aborted) ** in thread 7f8bcebec700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55be2147f120] 2: (()+0x14120) [0x7f8bde1ad120] 3: (gsignal()+0x110) [0x7f8bdd05efc0] 4: (abort()+0x1c7) [0x7f8bdd060b07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f8bdfc51df0] 6: (()+0x283e67) [0x7f8bdfc51e67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55be20fafefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55be20f63ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55be20f6cd41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55be20f6d066] 11: (DispatchQueue::entry()+0xb92) [0x7f8bdfcca4a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f8bdfd67dad] 13: (()+0x792a) [0x7f8bde1a092a] 14: (clone()+0x3f) [0x7f8bdd12b87f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:19:15.217 7f050af06280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:19:15.217 7f050af06280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1376 2018-07-25 11:19:15.217 7f050af06280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:19:15.227 7f050af06280 -1 asok(0x5557acffc240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:19:15.267 7f050af06280 0 load: jerasure load: lrc load: isa 2018-07-25 11:19:15.287 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:15.287 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:19:15.617 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:19:15.617 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:15.617 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:19:15.617 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:15.637 7f050af06280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:19:15.637 7f050af06280 1 bluefs mount 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:19:15.687 7f050af06280 1 rocksdb: do_open column families: [default] 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: DB SUMMARY 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: MANIFEST file: MANIFEST-000020 size: 374 Bytes 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: SST files in db dir, Total Num: 5, files: 000004.sst 000011.sst 000015.sst 000019.sst 000024.sst 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Write Ahead Log file in db: 000021.log size: 0 ; 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.env: 0x5557ad3213e0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.info_log: 0x5557ad343660 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_buffer_manager: 0x5557ad33fc20 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.row_cache: None 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000020 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5557acfc53c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5557ad077490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.ttl: 0 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000020 succeeded,manifest_file_number is 20, next_file_number is 26, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 17 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 18 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555757676, "job": 1, "event": "recovery_started", "log_files": [21]} 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #21 mode 0 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 26 2018-07-25 11:19:15.807 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555815882, "job": 1, "event": "recovery_finished"} 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832649, "job": 2, "event": "table_file_deletion", "file_number": 19} 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832847, "job": 2, "event": "table_file_deletion", "file_number": 15} 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832972, "job": 2, "event": "table_file_deletion", "file_number": 11} 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555833066, "job": 2, "event": "table_file_deletion", "file_number": 4} 2018-07-25 11:19:15.847 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x5557ad449000 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:19:15.847 7f050af06280 1 freelist init 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:19:15.857 7f050af06280 0 _get_class not permitted to load kvs 2018-07-25 11:19:15.857 7f050af06280 0 _get_class not permitted to load lua 2018-07-25 11:19:15.857 7f050af06280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:19:15.937 7f050af06280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:19:15.937 7f050af06280 0 _get_class not permitted to load sdk 2018-07-25 11:19:15.937 7f050af06280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 load_pgs 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:19:15.967 7f050af06280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:19:16.137 7f050af06280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:19:16.137 7f050af06280 1 osd.7 0 start_boot 2018-07-25 11:19:16.147 7f04e76aa700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:19:16.167 7f04f1309700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:19:16.167 7f04f1309700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f04f1309700 time 2018-07-25 11:19:16.175310 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f050236eca2] 2: (()+0x283e67) [0x7f050236ee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x5557aacbcefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5557aac70ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x5557aac79d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x5557aac7a066] 7: (DispatchQueue::entry()+0xb92) [0x7f05023e74a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0502484dad] 9: (()+0x792a) [0x7f05008bd92a] 10: (clone()+0x3f) [0x7f04ff84887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -458> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perfcounters_dump hook 0x5557acfc6060 -457> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command 1 hook 0x5557acfc6060 -456> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perf dump hook 0x5557acfc6060 -455> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perfcounters_schema hook 0x5557acfc6060 -454> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perf histogram dump hook 0x5557acfc6060 -453> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command 2 hook 0x5557acfc6060 -452> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perf schema hook 0x5557acfc6060 -451> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perf histogram schema hook 0x5557acfc6060 -450> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command perf reset hook 0x5557acfc6060 -449> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config show hook 0x5557acfc6060 -448> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config help hook 0x5557acfc6060 -447> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config set hook 0x5557acfc6060 -446> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config unset hook 0x5557acfc6060 -445> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config get hook 0x5557acfc6060 -444> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config diff hook 0x5557acfc6060 -443> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command config diff get hook 0x5557acfc6060 -442> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command log flush hook 0x5557acfc6060 -441> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command log dump hook 0x5557acfc6060 -440> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command log reopen hook 0x5557acfc6060 -439> 2018-07-25 11:19:15.157 7f050af06280 5 asok(0x5557acffc240) register_command dump_mempools hook 0x5557ad348068 -438> 2018-07-25 11:19:15.177 7f050af06280 10 monclient: get_monmap_and_config -437> 2018-07-25 11:19:15.177 7f050af06280 10 monclient: build_initial_monmap -436> 2018-07-25 11:19:15.197 7f04fc197700 2 Event(0x5557acffd4c0 nevent=5000 time_id=1).set_owner idx=2 owner=139659386124032 -435> 2018-07-25 11:19:15.197 7f04fc998700 2 Event(0x5557acffd280 nevent=5000 time_id=1).set_owner idx=1 owner=139659394516736 -434> 2018-07-25 11:19:15.197 7f04fd199700 2 Event(0x5557acffd040 nevent=5000 time_id=1).set_owner idx=0 owner=139659402909440 -433> 2018-07-25 11:19:15.197 7f050af06280 1 Processor -- start -432> 2018-07-25 11:19:15.197 7f050af06280 1 -- - start start -431> 2018-07-25 11:19:15.197 7f050af06280 10 monclient: init -430> 2018-07-25 11:19:15.197 7f050af06280 5 adding auth protocol: cephx -429> 2018-07-25 11:19:15.197 7f050af06280 10 monclient: auth_supported 2 method cephx -428> 2018-07-25 11:19:15.197 7f050af06280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -427> 2018-07-25 11:19:15.197 7f050af06280 10 monclient: _reopen_session rank -1 -426> 2018-07-25 11:19:15.197 7f050af06280 10 monclient(hunting): picked mon.mimir con 0x5557ad426700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -425> 2018-07-25 11:19:15.197 7f050af06280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5557ad396780 con 0 -424> 2018-07-25 11:19:15.197 7f050af06280 10 monclient(hunting): _renew_subs -423> 2018-07-25 11:19:15.197 7f050af06280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:24:15.207062 -422> 2018-07-25 11:19:15.197 7f04fc998700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 -421> 2018-07-25 11:19:15.197 7f04fc998700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -420> 2018-07-25 11:19:15.197 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 1 0x5557ad438000 mon_map magic: 0 v1 -419> 2018-07-25 11:19:15.197 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 2 0x5557ad396780 auth_reply(proto 2 0 (0) Success) v1 -418> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5557ad438000 con 0x5557ad426700 -417> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -416> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -415> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -414> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2663537942 0 0) 0x5557ad396780 con 0x5557ad426700 -413> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient(hunting): my global_id is 220358 -412> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5557ad396f00 con 0 -411> 2018-07-25 11:19:15.197 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 3 0x5557ad396f00 auth_reply(proto 2 0 (0) Success) v1 -410> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3295892916 0 0) 0x5557ad396f00 con 0x5557ad426700 -409> 2018-07-25 11:19:15.197 7f04fb996700 1 monclient: found mon.mimir -408> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -407> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557acffd680 con 0 -406> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:18:45.213142) -405> 2018-07-25 11:19:15.197 7f04fb996700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -404> 2018-07-25 11:19:15.197 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5557ad396780 con 0 -403> 2018-07-25 11:19:15.197 7f050af06280 5 monclient: authenticate success, global_id 220358 -402> 2018-07-25 11:19:15.207 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 4 0x5557acffd680 config(0 keys) v1 -401> 2018-07-25 11:19:15.207 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 5 0x5557ad438480 mon_map magic: 0 v1 -400> 2018-07-25 11:19:15.207 7f04fb996700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x5557acffd680 con 0x5557ad426700 -399> 2018-07-25 11:19:15.207 7f04fb996700 10 monclient: handle_config config(0 keys) v1 -398> 2018-07-25 11:19:15.207 7f04fa193700 4 set_mon_vals no callback set -397> 2018-07-25 11:19:15.207 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62454 cs=1 l=1). rx mon.2 seq 6 0x5557ad396780 auth_reply(proto 2 0 (0) Success) v1 -396> 2018-07-25 11:19:15.207 7f050af06280 10 monclient: get_monmap_and_config success -395> 2018-07-25 11:19:15.207 7f050af06280 10 monclient: shutdown -394> 2018-07-25 11:19:15.207 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN pgs=62454 cs=1 l=1).mark_down -393> 2018-07-25 11:19:15.207 7f050af06280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad426700 :-1 s=STATE_OPEN pgs=62454 cs=1 l=1)._stop -392> 2018-07-25 11:19:15.207 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 shutdown_connections -391> 2018-07-25 11:19:15.207 7f050af06280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x5557ad426700 -390> 2018-07-25 11:19:15.207 7f050af06280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 shutdown_connections delete 0x5557ad426700 -389> 2018-07-25 11:19:15.207 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 shutdown_connections -388> 2018-07-25 11:19:15.207 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 wait complete. -387> 2018-07-25 11:19:15.207 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 conn(0x5557ad426000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -386> 2018-07-25 11:19:15.207 7f050af06280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/3298506085 conn(0x5557ad426000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -385> 2018-07-25 11:19:15.217 7f050af06280 0 set uid:gid to 114:993 (ceph:ceph) -384> 2018-07-25 11:19:15.217 7f050af06280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1376 -383> 2018-07-25 11:19:15.217 7f050af06280 0 pidfile_write: ignore empty --pid-file -382> 2018-07-25 11:19:15.217 7f050af06280 5 asok(0x5557acffc240) init /var/run/ceph/ceph-osd.7.asok -381> 2018-07-25 11:19:15.227 7f050af06280 5 asok(0x5557acffc240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -380> 2018-07-25 11:19:15.227 7f050af06280 -1 asok(0x5557acffc240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -379> 2018-07-25 11:19:15.227 7f050af06280 5 object store type is bluestore -378> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -377> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 -376> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -375> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1376 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1376 -374> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -373> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1376 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1376 -372> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -371> 2018-07-25 11:19:15.247 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1376 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1376 -370> 2018-07-25 11:19:15.247 7f050af06280 10 monclient: build_initial_monmap -369> 2018-07-25 11:19:15.267 7f050af06280 0 load: jerasure load: lrc load: isa -368> 2018-07-25 11:19:15.287 7f050af06280 5 adding auth protocol: cephx -367> 2018-07-25 11:19:15.287 7f050af06280 5 adding auth protocol: cephx -366> 2018-07-25 11:19:15.287 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -365> 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -364> 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -363> 2018-07-25 11:19:15.287 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -362> 2018-07-25 11:19:15.287 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) close -361> 2018-07-25 11:19:15.617 7f050af06280 5 asok(0x5557acffc240) register_command objecter_requests hook 0x5557acfc6650 -360> 2018-07-25 11:19:15.617 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 start start -359> 2018-07-25 11:19:15.617 7f050af06280 1 -- - start start -358> 2018-07-25 11:19:15.617 7f050af06280 1 -- - start start -357> 2018-07-25 11:19:15.617 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1376 start start -356> 2018-07-25 11:19:15.617 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1376 start start -355> 2018-07-25 11:19:15.617 7f050af06280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1376 start start -354> 2018-07-25 11:19:15.617 7f050af06280 1 -- - start start -353> 2018-07-25 11:19:15.617 7f050af06280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -352> 2018-07-25 11:19:15.617 7f050af06280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -351> 2018-07-25 11:19:15.617 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -350> 2018-07-25 11:19:15.617 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -349> 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -348> 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -347> 2018-07-25 11:19:15.617 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -346> 2018-07-25 11:19:15.617 7f050af06280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -345> 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -344> 2018-07-25 11:19:15.617 7f050af06280 1 bdev(0x5557ad4fa380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -343> 2018-07-25 11:19:15.637 7f050af06280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -342> 2018-07-25 11:19:15.637 7f050af06280 1 bluefs mount -341> 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option compaction_readahead_size = 2097152 -340> 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option compression = kNoCompression -339> 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option max_write_buffer_number = 4 -338> 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -337> 2018-07-25 11:19:15.667 7f050af06280 0 set rocksdb option recycle_log_file_num = 4 -336> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option writable_file_max_buffer_size = 0 -335> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option write_buffer_size = 268435456 -334> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option compaction_readahead_size = 2097152 -333> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option compression = kNoCompression -332> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option max_write_buffer_number = 4 -331> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -330> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option recycle_log_file_num = 4 -329> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option writable_file_max_buffer_size = 0 -328> 2018-07-25 11:19:15.677 7f050af06280 0 set rocksdb option write_buffer_size = 268435456 -327> 2018-07-25 11:19:15.687 7f050af06280 1 rocksdb: do_open column families: [default] -326> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: RocksDB version: 5.13.0 -325> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -324> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Compile date Jul 24 2018 -323> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: DB SUMMARY -322> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: CURRENT file: CURRENT -321> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: IDENTITY file: IDENTITY -320> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: MANIFEST file: MANIFEST-000020 size: 374 Bytes -319> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: SST files in db dir, Total Num: 5, files: 000004.sst 000011.sst 000015.sst 000019.sst 000024.sst -318> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Write Ahead Log file in db: 000021.log size: 0 ; -317> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.error_if_exists: 0 -316> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.create_if_missing: 0 -315> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.paranoid_checks: 1 -314> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.env: 0x5557ad3213e0 -313> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.info_log: 0x5557ad343660 -312> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_file_opening_threads: 16 -311> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.statistics: (nil) -310> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_fsync: 0 -309> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_log_file_size: 0 -308> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -307> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.log_file_time_to_roll: 0 -306> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.keep_log_file_num: 1000 -305> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.recycle_log_file_num: 4 -304> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_fallocate: 1 -303> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_mmap_reads: 0 -302> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_mmap_writes: 0 -301> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_direct_reads: 0 -300> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -299> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.create_missing_column_families: 0 -298> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.db_log_dir: -297> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_dir: db -296> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.table_cache_numshardbits: 6 -295> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_subcompactions: 1 -294> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_flushes: -1 -293> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.WAL_ttl_seconds: 0 -292> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.WAL_size_limit_MB: 0 -291> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -290> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.is_fd_close_on_exec: 1 -289> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.advise_random_on_open: 1 -288> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.db_write_buffer_size: 0 -287> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_buffer_manager: 0x5557ad33fc20 -286> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -285> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -284> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -283> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.use_adaptive_mutex: 0 -282> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.rate_limiter: (nil) -281> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -280> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_recovery_mode: 0 -279> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_thread_tracking: 0 -278> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_pipelined_write: 0 -277> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -276> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -275> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -274> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -273> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.row_cache: None -272> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_filter: None -271> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -270> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.allow_ingest_behind: 0 -269> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.preserve_deletes: 0 -268> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.two_write_queues: 0 -267> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.manual_wal_flush: 0 -266> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_jobs: 2 -265> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_background_compactions: -1 -264> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -263> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -262> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.delayed_write_rate : 16777216 -261> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_total_wal_size: 0 -260> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -259> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.stats_dump_period_sec: 600 -258> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_open_files: -1 -257> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.bytes_per_sync: 0 -256> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.wal_bytes_per_sync: 0 -255> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_readahead_size: 2097152 -254> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Compression algorithms supported: -253> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -252> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZSTD supported: 0 -251> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kXpressCompression supported: 0 -250> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kLZ4HCCompression supported: 1 -249> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kLZ4Compression supported: 1 -248> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kBZip2Compression supported: 0 -247> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kZlibCompression supported: 1 -246> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: kSnappyCompression supported: 1 -245> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -244> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000020 -243> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -242> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -241> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -240> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_filter: None -239> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compaction_filter_factory: None -238> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.memtable_factory: SkipListFactory -237> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.table_factory: BlockBasedTable -236> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5557acfc53c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5557ad077490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -235> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.write_buffer_size: 268435456 -234> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_write_buffer_number: 4 -233> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.compression: NoCompression -232> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.bottommost_compression: Disabled -231> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.prefix_extractor: nullptr -230> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -229> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.num_levels: 7 -228> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -227> 2018-07-25 11:19:15.687 7f050af06280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -226> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.window_bits: -14 -225> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.level: -1 -224> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.strategy: 0 -223> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -222> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -221> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -220> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -219> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -218> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.target_file_size_base: 67108864 -217> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.target_file_size_multiplier: 1 -216> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -215> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -214> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -213> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -212> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -211> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -210> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -209> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -208> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -207> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -206> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -205> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -204> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.arena_block_size: 33554432 -203> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -202> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -201> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -200> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.disable_auto_compactions: 0 -199> 2018-07-25 11:19:15.697 7f050af06280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -198> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -197> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -196> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -195> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -194> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -193> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -192> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -191> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -190> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -189> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -188> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.table_properties_collectors: -187> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.inplace_update_support: 0 -186> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.inplace_update_num_locks: 10000 -185> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -184> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.memtable_huge_page_size: 0 -183> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.bloom_locality: 0 -182> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.max_successive_merges: 0 -181> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.optimize_filters_for_hits: 0 -180> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.paranoid_file_checks: 0 -179> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.force_consistency_checks: 0 -178> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.report_bg_io_stats: 0 -177> 2018-07-25 11:19:15.707 7f050af06280 4 rocksdb: Options.ttl: 0 -176> 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000020 succeeded,manifest_file_number is 20, next_file_number is 26, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 17 -175> 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 18 -174> 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555757676, "job": 1, "event": "recovery_started", "log_files": [21]} -173> 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #21 mode 0 -172> 2018-07-25 11:19:15.747 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 26 -171> 2018-07-25 11:19:15.807 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555815882, "job": 1, "event": "recovery_finished"} -170> 2018-07-25 11:19:15.807 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000020 type=3 #20 -- OK -169> 2018-07-25 11:19:15.817 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000021.log type=0 #21 -- OK -168> 2018-07-25 11:19:15.817 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db/000019.sst type=2 #19 -- OK -167> 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832649, "job": 2, "event": "table_file_deletion", "file_number": 19} -166> 2018-07-25 11:19:15.817 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db/000015.sst type=2 #15 -- OK -165> 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832847, "job": 2, "event": "table_file_deletion", "file_number": 15} -164> 2018-07-25 11:19:15.817 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db/000011.sst type=2 #11 -- OK -163> 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555832972, "job": 2, "event": "table_file_deletion", "file_number": 11} -162> 2018-07-25 11:19:15.817 7f050af06280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db/000004.sst type=2 #4 -- OK -161> 2018-07-25 11:19:15.817 7f050af06280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535555833066, "job": 2, "event": "table_file_deletion", "file_number": 4} -160> 2018-07-25 11:19:15.847 7f050af06280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x5557ad449000 -159> 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -158> 2018-07-25 11:19:15.847 7f050af06280 1 freelist init -157> 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -156> 2018-07-25 11:19:15.847 7f050af06280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -155> 2018-07-25 11:19:15.847 7f050af06280 2 osd.7 0 journal looks like hdd -154> 2018-07-25 11:19:15.847 7f050af06280 2 osd.7 0 boot -153> 2018-07-25 11:19:15.857 7f050af06280 0 _get_class not permitted to load kvs -152> 2018-07-25 11:19:15.857 7f050af06280 0 _get_class not permitted to load lua -151> 2018-07-25 11:19:15.857 7f050af06280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -150> 2018-07-25 11:19:15.857 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -149> 2018-07-25 11:19:15.917 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -148> 2018-07-25 11:19:15.917 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -147> 2018-07-25 11:19:15.917 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -146> 2018-07-25 11:19:15.917 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -145> 2018-07-25 11:19:15.937 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -144> 2018-07-25 11:19:15.937 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -143> 2018-07-25 11:19:15.937 7f050af06280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -142> 2018-07-25 11:19:15.937 7f050af06280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -141> 2018-07-25 11:19:15.937 7f050af06280 0 _get_class not permitted to load sdk -140> 2018-07-25 11:19:15.937 7f050af06280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -139> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -138> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -137> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -136> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 load_pgs -135> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 load_pgs opened 0 pgs -134> 2018-07-25 11:19:15.937 7f050af06280 2 osd.7 0 superblock: I am osd.7 -133> 2018-07-25 11:19:15.937 7f050af06280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -132> 2018-07-25 11:19:15.937 7f050af06280 1 Processor -- start -131> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -130> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -129> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -128> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -127> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -126> 2018-07-25 11:19:15.967 7f050af06280 1 Processor -- start -125> 2018-07-25 11:19:15.967 7f050af06280 10 monclient: init -124> 2018-07-25 11:19:15.967 7f050af06280 5 adding auth protocol: cephx -123> 2018-07-25 11:19:15.967 7f050af06280 10 monclient: auth_supported 2 method cephx -122> 2018-07-25 11:19:15.967 7f050af06280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -121> 2018-07-25 11:19:15.967 7f050af06280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -120> 2018-07-25 11:19:15.967 7f050af06280 -1 osd.7 0 log_to_monitors {default=true} -119> 2018-07-25 11:19:15.967 7f050af06280 10 monclient: _reopen_session rank -1 -118> 2018-07-25 11:19:15.967 7f050af06280 10 monclient(hunting): picked mon.mimir con 0x5557ad429800 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -117> 2018-07-25 11:19:15.967 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5557ad397900 con 0 -116> 2018-07-25 11:19:15.967 7f050af06280 10 monclient(hunting): _renew_subs -115> 2018-07-25 11:19:15.977 7f04dfe9b700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -114> 2018-07-25 11:19:15.987 7f04fc998700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -113> 2018-07-25 11:19:15.997 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 1 0x5557ad438900 mon_map magic: 0 v1 -112> 2018-07-25 11:19:15.997 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 2 0x5557ad397900 auth_reply(proto 2 0 (0) Success) v1 -111> 2018-07-25 11:19:16.007 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5557ad438900 con 0x5557ad429800 -110> 2018-07-25 11:19:16.007 7f04f1309700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -109> 2018-07-25 11:19:16.007 7f04f1309700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -108> 2018-07-25 11:19:16.007 7f04f1309700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -107> 2018-07-25 11:19:16.007 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2076322436 0 0) 0x5557ad397900 con 0x5557ad429800 -106> 2018-07-25 11:19:16.007 7f04f1309700 10 monclient(hunting): my global_id is 220361 -105> 2018-07-25 11:19:16.007 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5557ad790000 con 0 -104> 2018-07-25 11:19:16.027 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 3 0x5557ad790000 auth_reply(proto 2 0 (0) Success) v1 -103> 2018-07-25 11:19:16.037 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2628861161 0 0) 0x5557ad790000 con 0x5557ad429800 -102> 2018-07-25 11:19:16.037 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5557ad397900 con 0 -101> 2018-07-25 11:19:16.047 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 4 0x5557ad397900 auth_reply(proto 2 0 (0) Success) v1 -100> 2018-07-25 11:19:16.057 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (1797831067 0 0) 0x5557ad397900 con 0x5557ad429800 -99> 2018-07-25 11:19:16.057 7f04f1309700 1 monclient: found mon.mimir -98> 2018-07-25 11:19:16.057 7f04f1309700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -97> 2018-07-25 11:19:16.057 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557ad438240 con 0 -96> 2018-07-25 11:19:16.057 7f04f1309700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:18:46.064397) -95> 2018-07-25 11:19:16.057 7f04f1309700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -94> 2018-07-25 11:19:16.057 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5557ad790000 con 0 -93> 2018-07-25 11:19:16.057 7f050af06280 5 monclient: authenticate success, global_id 220361 -92> 2018-07-25 11:19:16.057 7f050af06280 10 monclient: wait_auth_rotating waiting (until 2018-07-25 11:19:46.064566) -91> 2018-07-25 11:19:16.067 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 5 0x5557ad438240 config(0 keys) v1 -90> 2018-07-25 11:19:16.067 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 6 0x5557ad78cfc0 mon_map magic: 0 v1 -89> 2018-07-25 11:19:16.067 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 7 0x5557ad790000 auth_reply(proto 2 0 (0) Success) v1 -88> 2018-07-25 11:19:16.077 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x5557ad438240 con 0x5557ad429800 -87> 2018-07-25 11:19:16.077 7f04f1309700 10 monclient: handle_config config(0 keys) v1 -86> 2018-07-25 11:19:16.077 7f04e76aa700 4 set_mon_vals no callback set -85> 2018-07-25 11:19:16.077 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x5557ad78cfc0 con 0x5557ad429800 -84> 2018-07-25 11:19:16.077 7f04f1309700 10 monclient: handle_monmap mon_map magic: 0 v1 -83> 2018-07-25 11:19:16.077 7f04f1309700 10 monclient: got monmap 3, mon.mimir is now rank 2 -82> 2018-07-25 11:19:16.077 7f04f1309700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -81> 2018-07-25 11:19:16.077 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (645904946 0 0) 0x5557ad790000 con 0x5557ad429800 -80> 2018-07-25 11:19:16.077 7f04f1309700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:18:46.086579) -79> 2018-07-25 11:19:16.077 7f050af06280 10 monclient: wait_auth_rotating done -78> 2018-07-25 11:19:16.077 7f050af06280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -77> 2018-07-25 11:19:16.077 7f050af06280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -76> 2018-07-25 11:19:16.077 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x5557ad439440 con 0 -75> 2018-07-25 11:19:16.127 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 8 0x5557ad439440 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 -74> 2018-07-25 11:19:16.127 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 ==== 207+0+0 (1704255731 0 0) 0x5557ad439440 con 0x5557ad429800 -73> 2018-07-25 11:19:16.127 7f04f1309700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -72> 2018-07-25 11:19:16.127 7f04f1309700 10 monclient: _finish_command 1 = 0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. -71> 2018-07-25 11:19:16.127 7f050af06280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -70> 2018-07-25 11:19:16.127 7f050af06280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -69> 2018-07-25 11:19:16.127 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x5557ad439680 con 0 -68> 2018-07-25 11:19:16.137 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 9 0x5557ad439680 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 -67> 2018-07-25 11:19:16.137 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 ==== 245+0+0 (3636428954 0 0) 0x5557ad439680 con 0x5557ad429800 -66> 2018-07-25 11:19:16.137 7f04f1309700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -65> 2018-07-25 11:19:16.137 7f04f1309700 10 monclient: _finish_command 2 = 0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -64> 2018-07-25 11:19:16.137 7f050af06280 0 osd.7 0 done with init, starting boot process -63> 2018-07-25 11:19:16.137 7f050af06280 10 monclient: _renew_subs -62> 2018-07-25 11:19:16.137 7f050af06280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -61> 2018-07-25 11:19:16.137 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x5557acffd8c0 con 0 -60> 2018-07-25 11:19:16.137 7f050af06280 1 osd.7 0 start_boot -59> 2018-07-25 11:19:16.137 7f050af06280 10 monclient: get_version osdmap req 0x5557ad7883a0 -58> 2018-07-25 11:19:16.137 7f050af06280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -57> 2018-07-25 11:19:16.137 7f050af06280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x5557ad521800 con 0 -56> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command status hook 0x5557acfc6820 -55> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command flush_journal hook 0x5557acfc6820 -54> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_ops_in_flight hook 0x5557acfc6820 -53> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command ops hook 0x5557acfc6820 -52> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_blocked_ops hook 0x5557acfc6820 -51> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_historic_ops hook 0x5557acfc6820 -50> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_historic_slow_ops hook 0x5557acfc6820 -49> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_historic_ops_by_duration hook 0x5557acfc6820 -48> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_op_pq_state hook 0x5557acfc6820 -47> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_blacklist hook 0x5557acfc6820 -46> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_watchers hook 0x5557acfc6820 -45> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_reservations hook 0x5557acfc6820 -44> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command get_latest_osdmap hook 0x5557acfc6820 -43> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command heap hook 0x5557acfc6820 -42> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command set_heap_property hook 0x5557acfc6820 -41> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command get_heap_property hook 0x5557acfc6820 -40> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_objectstore_kv_stats hook 0x5557acfc6820 -39> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_scrubs hook 0x5557acfc6820 -38> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command calc_objectstore_db_histogram hook 0x5557acfc6820 -37> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command flush_store_cache hook 0x5557acfc6820 -36> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command dump_pgstate_history hook 0x5557acfc6820 -35> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command compact hook 0x5557acfc6820 -34> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command get_mapped_pools hook 0x5557acfc6820 -33> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command smart hook 0x5557acfc6820 -32> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command list_devices hook 0x5557acfc6820 -31> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command setomapval hook 0x5557ad788a80 -30> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command rmomapkey hook 0x5557ad788a80 -29> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command setomapheader hook 0x5557ad788a80 -28> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command getomap hook 0x5557ad788a80 -27> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command truncobj hook 0x5557ad788a80 -26> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command injectdataerr hook 0x5557ad788a80 -25> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command injectmdataerr hook 0x5557ad788a80 -24> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command set_recovery_delay hook 0x5557ad788a80 -23> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command trigger_scrub hook 0x5557ad788a80 -22> 2018-07-25 11:19:16.137 7f050af06280 5 asok(0x5557acffc240) register_command injectfull hook 0x5557ad788a80 -21> 2018-07-25 11:19:16.137 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 10 0x5557ad545800 mgrmap(e 34) v1 -20> 2018-07-25 11:19:16.137 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 11 0x5557ad436000 mon_get_version_reply(handle=1 version=2922) v2 -19> 2018-07-25 11:19:16.137 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x5557ad545800 con 0x5557ad429800 -18> 2018-07-25 11:19:16.137 7f04f1309700 4 mgrc handle_mgr_map Got map version 34 -17> 2018-07-25 11:19:16.137 7f04f1309700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -16> 2018-07-25 11:19:16.137 7f04f1309700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -15> 2018-07-25 11:19:16.147 7f04fc197700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x5557ad660000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -14> 2018-07-25 11:19:16.147 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x5557ad7b0000 con 0 -13> 2018-07-25 11:19:16.147 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2922) v2 ==== 24+0+0 (2719405947 0 0) 0x5557ad436000 con 0x5557ad429800 -12> 2018-07-25 11:19:16.147 7f04f1309700 10 monclient: handle_get_version_reply finishing 0x5557ad7883a0 version 2922 -11> 2018-07-25 11:19:16.147 7f04e76aa700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -10> 2018-07-25 11:19:16.147 7f04e76aa700 -1 osd.7 0 waiting for initial osdmap -9> 2018-07-25 11:19:16.147 7f04e76aa700 10 monclient: _renew_subs -8> 2018-07-25 11:19:16.147 7f04e76aa700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -7> 2018-07-25 11:19:16.147 7f04e76aa700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x5557ad78d8c0 con 0 -6> 2018-07-25 11:19:16.147 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 12 0x5557ad79c2c0 osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 -5> 2018-07-25 11:19:16.147 7f04fc998700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x5557ad429800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62455 cs=1 l=1). rx mon.2 seq 13 0x5557ad79c580 osd_map(2320..2359 src has 2319..2922) v4 -4> 2018-07-25 11:19:16.167 7f04fc197700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x5557ad660000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339632 cs=1 l=1). rx mgr.34109 seq 1 0x5557ad692000 mgrconfigure(period=5, threshold=5) v2 -3> 2018-07-25 11:19:16.167 7f04f1309700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1376 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 ==== 4888+0+0 (2709902744 0 0) 0x5557ad79c2c0 con 0x5557ad429800 -2> 2018-07-25 11:19:16.167 7f04f1309700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2922] -1> 2018-07-25 11:19:16.167 7f04f1309700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 0> 2018-07-25 11:19:16.167 7f04f1309700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f04f1309700 time 2018-07-25 11:19:16.175310 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f050236eca2] 2: (()+0x283e67) [0x7f050236ee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x5557aacbcefa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5557aac70ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x5557aac79d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x5557aac7a066] 7: (DispatchQueue::entry()+0xb92) [0x7f05023e74a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0502484dad] 9: (()+0x792a) [0x7f05008bd92a] 10: (clone()+0x3f) [0x7f04ff84887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:19:16.177 7f04f1309700 -1 *** Caught signal (Aborted) ** in thread 7f04f1309700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x5557ab18c120] 2: (()+0x14120) [0x7f05008ca120] 3: (gsignal()+0x110) [0x7f04ff77bfc0] 4: (abort()+0x1c7) [0x7f04ff77db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f050236edf0] 6: (()+0x283e67) [0x7f050236ee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x5557aacbcefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5557aac70ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x5557aac79d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x5557aac7a066] 11: (DispatchQueue::entry()+0xb92) [0x7f05023e74a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0502484dad] 13: (()+0x792a) [0x7f05008bd92a] 14: (clone()+0x3f) [0x7f04ff84887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:19:16.177 7f04f1309700 -1 *** Caught signal (Aborted) ** in thread 7f04f1309700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x5557ab18c120] 2: (()+0x14120) [0x7f05008ca120] 3: (gsignal()+0x110) [0x7f04ff77bfc0] 4: (abort()+0x1c7) [0x7f04ff77db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f050236edf0] 6: (()+0x283e67) [0x7f050236ee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x5557aacbcefa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x5557aac70ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x5557aac79d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x5557aac7a066] 11: (DispatchQueue::entry()+0xb92) [0x7f05023e74a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0502484dad] 13: (()+0x792a) [0x7f05008bd92a] 14: (clone()+0x3f) [0x7f04ff84887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:19:41.787 7f2c18096280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:19:41.787 7f2c18096280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1472 2018-07-25 11:19:41.787 7f2c18096280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:19:41.787 7f2c18096280 -1 asok(0x55bb7b680240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:19:41.827 7f2c18096280 0 load: jerasure load: lrc load: isa 2018-07-25 11:19:41.847 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:41.847 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:19:42.177 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:19:42.177 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:42.177 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:42.177 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:42.217 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:19:42.217 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:19:42.217 7f2c18096280 1 bdev(0x55bb7bb7e380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:19:42.217 7f2c18096280 1 bdev(0x55bb7bb7e380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:19:42.217 7f2c18096280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:19:42.217 7f2c18096280 1 bluefs mount 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:19:42.327 7f2c18096280 1 rocksdb: do_open column families: [default] 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: DB SUMMARY 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: MANIFEST file: MANIFEST-000026 size: 168 Bytes 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000024.sst 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Write Ahead Log file in db: 000027.log size: 0 ; 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.env: 0x55bb7b9a4cc0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.info_log: 0x55bb7b9c7620 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.write_buffer_manager: 0x55bb7b9c3c20 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.row_cache: None 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000026 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bb7b6493c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55bb7b6fb490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.ttl: 0 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000026 succeeded,manifest_file_number is 26, next_file_number is 29, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 21 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 22 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535582389736, "job": 1, "event": "recovery_started", "log_files": [27]} 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #27 mode 0 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 29 2018-07-25 11:19:42.417 7f2c18096280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535582431359, "job": 1, "event": "recovery_finished"} 2018-07-25 11:19:42.457 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55bb7bacd000 2018-07-25 11:19:42.457 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:19:42.467 7f2c18096280 1 freelist init 2018-07-25 11:19:42.467 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:19:42.467 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:19:42.477 7f2c18096280 0 _get_class not permitted to load kvs 2018-07-25 11:19:42.477 7f2c18096280 0 _get_class not permitted to load lua 2018-07-25 11:19:42.477 7f2c18096280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:19:42.517 7f2c18096280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:19:42.517 7f2c18096280 0 _get_class not permitted to load sdk 2018-07-25 11:19:42.517 7f2c18096280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 load_pgs 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:19:42.547 7f2c18096280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:19:42.677 7f2c18096280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:19:42.677 7f2c18096280 1 osd.7 0 start_boot 2018-07-25 11:19:42.697 7f2bf483a700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:19:42.697 7f2bfe499700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:19:42.697 7f2bfe499700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f2bfe499700 time 2018-07-25 11:19:42.707508 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f2c0f4feca2] 2: (()+0x283e67) [0x7f2c0f4fee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55bb796a9efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55bb7965dff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55bb79666d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55bb79667066] 7: (DispatchQueue::entry()+0xb92) [0x7f2c0f5774a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2c0f614dad] 9: (()+0x792a) [0x7f2c0da4d92a] 10: (clone()+0x3f) [0x7f2c0c9d887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -450> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perfcounters_dump hook 0x55bb7b64a060 -449> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command 1 hook 0x55bb7b64a060 -448> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perf dump hook 0x55bb7b64a060 -447> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perfcounters_schema hook 0x55bb7b64a060 -446> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perf histogram dump hook 0x55bb7b64a060 -445> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command 2 hook 0x55bb7b64a060 -444> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perf schema hook 0x55bb7b64a060 -443> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perf histogram schema hook 0x55bb7b64a060 -442> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command perf reset hook 0x55bb7b64a060 -441> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config show hook 0x55bb7b64a060 -440> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config help hook 0x55bb7b64a060 -439> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config set hook 0x55bb7b64a060 -438> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config unset hook 0x55bb7b64a060 -437> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config get hook 0x55bb7b64a060 -436> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config diff hook 0x55bb7b64a060 -435> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command config diff get hook 0x55bb7b64a060 -434> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command log flush hook 0x55bb7b64a060 -433> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command log dump hook 0x55bb7b64a060 -432> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command log reopen hook 0x55bb7b64a060 -431> 2018-07-25 11:19:41.707 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_mempools hook 0x55bb7b9cc068 -430> 2018-07-25 11:19:41.727 7f2c18096280 10 monclient: get_monmap_and_config -429> 2018-07-25 11:19:41.737 7f2c18096280 10 monclient: build_initial_monmap -428> 2018-07-25 11:19:41.757 7f2c09327700 2 Event(0x55bb7b6814c0 nevent=5000 time_id=1).set_owner idx=2 owner=139827109590784 -427> 2018-07-25 11:19:41.757 7f2c09b28700 2 Event(0x55bb7b681280 nevent=5000 time_id=1).set_owner idx=1 owner=139827117983488 -426> 2018-07-25 11:19:41.757 7f2c0a329700 2 Event(0x55bb7b681040 nevent=5000 time_id=1).set_owner idx=0 owner=139827126376192 -425> 2018-07-25 11:19:41.757 7f2c18096280 1 Processor -- start -424> 2018-07-25 11:19:41.757 7f2c18096280 1 -- - start start -423> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient: init -422> 2018-07-25 11:19:41.757 7f2c18096280 5 adding auth protocol: cephx -421> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient: auth_supported 2 method cephx -420> 2018-07-25 11:19:41.757 7f2c18096280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -419> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient: _reopen_session rank -1 -418> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient(hunting): picked mon.mimir con 0x55bb7baaa700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -417> 2018-07-25 11:19:41.757 7f2c18096280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55bb7ba1a780 con 0 -416> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient(hunting): _renew_subs -415> 2018-07-25 11:19:41.757 7f2c18096280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:24:41.767052 -414> 2018-07-25 11:19:41.757 7f2c09b28700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 -413> 2018-07-25 11:19:41.757 7f2c09b28700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -412> 2018-07-25 11:19:41.757 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 1 0x55bb7babc000 mon_map magic: 0 v1 -411> 2018-07-25 11:19:41.757 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 2 0x55bb7ba1a780 auth_reply(proto 2 0 (0) Success) v1 -410> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55bb7babc000 con 0x55bb7baaa700 -409> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -408> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -407> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -406> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1628038202 0 0) 0x55bb7ba1a780 con 0x55bb7baaa700 -405> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient(hunting): my global_id is 220364 -404> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55bb7ba1af00 con 0 -403> 2018-07-25 11:19:41.757 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 3 0x55bb7ba1af00 auth_reply(proto 2 0 (0) Success) v1 -402> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (925705509 0 0) 0x55bb7ba1af00 con 0x55bb7baaa700 -401> 2018-07-25 11:19:41.757 7f2c08b26700 1 monclient: found mon.mimir -400> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -399> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb7b681680 con 0 -398> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:19:11.773349) -397> 2018-07-25 11:19:41.757 7f2c08b26700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -396> 2018-07-25 11:19:41.757 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55bb7ba1a780 con 0 -395> 2018-07-25 11:19:41.757 7f2c18096280 5 monclient: authenticate success, global_id 220364 -394> 2018-07-25 11:19:41.767 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 4 0x55bb7b681680 config(0 keys) v1 -393> 2018-07-25 11:19:41.767 7f2c08b26700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55bb7b681680 con 0x55bb7baaa700 -392> 2018-07-25 11:19:41.767 7f2c08b26700 10 monclient: handle_config config(0 keys) v1 -391> 2018-07-25 11:19:41.767 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 5 0x55bb7babc480 mon_map magic: 0 v1 -390> 2018-07-25 11:19:41.767 7f2c07323700 4 set_mon_vals no callback set -389> 2018-07-25 11:19:41.767 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62456 cs=1 l=1). rx mon.2 seq 6 0x55bb7ba1a780 auth_reply(proto 2 0 (0) Success) v1 -388> 2018-07-25 11:19:41.767 7f2c18096280 10 monclient: get_monmap_and_config success -387> 2018-07-25 11:19:41.767 7f2c18096280 10 monclient: shutdown -386> 2018-07-25 11:19:41.767 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN pgs=62456 cs=1 l=1).mark_down -385> 2018-07-25 11:19:41.767 7f2c18096280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baaa700 :-1 s=STATE_OPEN pgs=62456 cs=1 l=1)._stop -384> 2018-07-25 11:19:41.767 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 shutdown_connections -383> 2018-07-25 11:19:41.767 7f2c18096280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x55bb7baaa700 -382> 2018-07-25 11:19:41.767 7f2c18096280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 shutdown_connections delete 0x55bb7baaa700 -381> 2018-07-25 11:19:41.767 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 shutdown_connections -380> 2018-07-25 11:19:41.767 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 wait complete. -379> 2018-07-25 11:19:41.767 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 conn(0x55bb7baaa000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -378> 2018-07-25 11:19:41.767 7f2c18096280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/304079372 conn(0x55bb7baaa000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -377> 2018-07-25 11:19:41.787 7f2c18096280 0 set uid:gid to 114:993 (ceph:ceph) -376> 2018-07-25 11:19:41.787 7f2c18096280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1472 -375> 2018-07-25 11:19:41.787 7f2c18096280 0 pidfile_write: ignore empty --pid-file -374> 2018-07-25 11:19:41.787 7f2c18096280 5 asok(0x55bb7b680240) init /var/run/ceph/ceph-osd.7.asok -373> 2018-07-25 11:19:41.787 7f2c18096280 5 asok(0x55bb7b680240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -372> 2018-07-25 11:19:41.787 7f2c18096280 -1 asok(0x55bb7b680240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -371> 2018-07-25 11:19:41.787 7f2c18096280 5 object store type is bluestore -370> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -369> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 -368> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -367> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1472 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1472 -366> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -365> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1472 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1472 -364> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -363> 2018-07-25 11:19:41.807 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1472 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1472 -362> 2018-07-25 11:19:41.807 7f2c18096280 10 monclient: build_initial_monmap -361> 2018-07-25 11:19:41.827 7f2c18096280 0 load: jerasure load: lrc load: isa -360> 2018-07-25 11:19:41.847 7f2c18096280 5 adding auth protocol: cephx -359> 2018-07-25 11:19:41.847 7f2c18096280 5 adding auth protocol: cephx -358> 2018-07-25 11:19:41.847 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -357> 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -356> 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -355> 2018-07-25 11:19:41.847 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -354> 2018-07-25 11:19:41.847 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) close -353> 2018-07-25 11:19:42.177 7f2c18096280 5 asok(0x55bb7b680240) register_command objecter_requests hook 0x55bb7b64a650 -352> 2018-07-25 11:19:42.177 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 start start -351> 2018-07-25 11:19:42.177 7f2c18096280 1 -- - start start -350> 2018-07-25 11:19:42.177 7f2c18096280 1 -- - start start -349> 2018-07-25 11:19:42.177 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1472 start start -348> 2018-07-25 11:19:42.177 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1472 start start -347> 2018-07-25 11:19:42.177 7f2c18096280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1472 start start -346> 2018-07-25 11:19:42.177 7f2c18096280 1 -- - start start -345> 2018-07-25 11:19:42.177 7f2c18096280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -344> 2018-07-25 11:19:42.177 7f2c18096280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -343> 2018-07-25 11:19:42.177 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -342> 2018-07-25 11:19:42.177 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -341> 2018-07-25 11:19:42.177 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -340> 2018-07-25 11:19:42.177 7f2c18096280 1 bdev(0x55bb7bb7e000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -339> 2018-07-25 11:19:42.217 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -338> 2018-07-25 11:19:42.217 7f2c18096280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -337> 2018-07-25 11:19:42.217 7f2c18096280 1 bdev(0x55bb7bb7e380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -336> 2018-07-25 11:19:42.217 7f2c18096280 1 bdev(0x55bb7bb7e380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -335> 2018-07-25 11:19:42.217 7f2c18096280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -334> 2018-07-25 11:19:42.217 7f2c18096280 1 bluefs mount -333> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compaction_readahead_size = 2097152 -332> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compression = kNoCompression -331> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option max_write_buffer_number = 4 -330> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -329> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option recycle_log_file_num = 4 -328> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option writable_file_max_buffer_size = 0 -327> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option write_buffer_size = 268435456 -326> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compaction_readahead_size = 2097152 -325> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option compression = kNoCompression -324> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option max_write_buffer_number = 4 -323> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -322> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option recycle_log_file_num = 4 -321> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option writable_file_max_buffer_size = 0 -320> 2018-07-25 11:19:42.277 7f2c18096280 0 set rocksdb option write_buffer_size = 268435456 -319> 2018-07-25 11:19:42.327 7f2c18096280 1 rocksdb: do_open column families: [default] -318> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: RocksDB version: 5.13.0 -317> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -316> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Compile date Jul 24 2018 -315> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: DB SUMMARY -314> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: CURRENT file: CURRENT -313> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: IDENTITY file: IDENTITY -312> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: MANIFEST file: MANIFEST-000026 size: 168 Bytes -311> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000024.sst -310> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Write Ahead Log file in db: 000027.log size: 0 ; -309> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.error_if_exists: 0 -308> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.create_if_missing: 0 -307> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.paranoid_checks: 1 -306> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.env: 0x55bb7b9a4cc0 -305> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.info_log: 0x55bb7b9c7620 -304> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_file_opening_threads: 16 -303> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.statistics: (nil) -302> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_fsync: 0 -301> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_log_file_size: 0 -300> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -299> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.log_file_time_to_roll: 0 -298> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.keep_log_file_num: 1000 -297> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.recycle_log_file_num: 4 -296> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_fallocate: 1 -295> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_mmap_reads: 0 -294> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.allow_mmap_writes: 0 -293> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_direct_reads: 0 -292> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -291> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.create_missing_column_families: 0 -290> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.db_log_dir: -289> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.wal_dir: db -288> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.table_cache_numshardbits: 6 -287> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_subcompactions: 1 -286> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.max_background_flushes: -1 -285> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.WAL_ttl_seconds: 0 -284> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.WAL_size_limit_MB: 0 -283> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -282> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.is_fd_close_on_exec: 1 -281> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.advise_random_on_open: 1 -280> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.db_write_buffer_size: 0 -279> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.write_buffer_manager: 0x55bb7b9c3c20 -278> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -277> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -276> 2018-07-25 11:19:42.327 7f2c18096280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -275> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.use_adaptive_mutex: 0 -274> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.rate_limiter: (nil) -273> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -272> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_recovery_mode: 0 -271> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_thread_tracking: 0 -270> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_pipelined_write: 0 -269> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -268> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -267> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -266> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -265> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.row_cache: None -264> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_filter: None -263> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -262> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.allow_ingest_behind: 0 -261> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.preserve_deletes: 0 -260> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.two_write_queues: 0 -259> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.manual_wal_flush: 0 -258> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_background_jobs: 2 -257> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_background_compactions: -1 -256> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -255> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -254> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.delayed_write_rate : 16777216 -253> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_total_wal_size: 0 -252> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -251> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.stats_dump_period_sec: 600 -250> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_open_files: -1 -249> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.bytes_per_sync: 0 -248> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.wal_bytes_per_sync: 0 -247> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_readahead_size: 2097152 -246> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Compression algorithms supported: -245> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -244> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZSTD supported: 0 -243> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kXpressCompression supported: 0 -242> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kLZ4HCCompression supported: 1 -241> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kLZ4Compression supported: 1 -240> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kBZip2Compression supported: 0 -239> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kZlibCompression supported: 1 -238> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: kSnappyCompression supported: 1 -237> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -236> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000026 -235> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -234> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -233> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -232> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_filter: None -231> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compaction_filter_factory: None -230> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.memtable_factory: SkipListFactory -229> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.table_factory: BlockBasedTable -228> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bb7b6493c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55bb7b6fb490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -227> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.write_buffer_size: 268435456 -226> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.max_write_buffer_number: 4 -225> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.compression: NoCompression -224> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.bottommost_compression: Disabled -223> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.prefix_extractor: nullptr -222> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -221> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.num_levels: 7 -220> 2018-07-25 11:19:42.337 7f2c18096280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -219> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -218> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.window_bits: -14 -217> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.level: -1 -216> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.strategy: 0 -215> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -214> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -213> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -212> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -211> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -210> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.target_file_size_base: 67108864 -209> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.target_file_size_multiplier: 1 -208> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -207> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -206> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -205> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -204> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -203> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -202> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -201> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -200> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -199> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -198> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -197> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -196> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.arena_block_size: 33554432 -195> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -194> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -193> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -192> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.disable_auto_compactions: 0 -191> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -190> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -189> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -188> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -187> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -186> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -185> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -184> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -183> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -182> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -181> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -180> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.table_properties_collectors: -179> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.inplace_update_support: 0 -178> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.inplace_update_num_locks: 10000 -177> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -176> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.memtable_huge_page_size: 0 -175> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.bloom_locality: 0 -174> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.max_successive_merges: 0 -173> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.optimize_filters_for_hits: 0 -172> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.paranoid_file_checks: 0 -171> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.force_consistency_checks: 0 -170> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.report_bg_io_stats: 0 -169> 2018-07-25 11:19:42.347 7f2c18096280 4 rocksdb: Options.ttl: 0 -168> 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000026 succeeded,manifest_file_number is 26, next_file_number is 29, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 21 -167> 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 22 -166> 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535582389736, "job": 1, "event": "recovery_started", "log_files": [27]} -165> 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #27 mode 0 -164> 2018-07-25 11:19:42.377 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 29 -163> 2018-07-25 11:19:42.417 7f2c18096280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535582431359, "job": 1, "event": "recovery_finished"} -162> 2018-07-25 11:19:42.417 7f2c18096280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000026 type=3 #26 -- OK -161> 2018-07-25 11:19:42.437 7f2c18096280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000027.log type=0 #27 -- OK -160> 2018-07-25 11:19:42.457 7f2c18096280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55bb7bacd000 -159> 2018-07-25 11:19:42.457 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -158> 2018-07-25 11:19:42.467 7f2c18096280 1 freelist init -157> 2018-07-25 11:19:42.467 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -156> 2018-07-25 11:19:42.467 7f2c18096280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -155> 2018-07-25 11:19:42.467 7f2c18096280 2 osd.7 0 journal looks like hdd -154> 2018-07-25 11:19:42.467 7f2c18096280 2 osd.7 0 boot -153> 2018-07-25 11:19:42.477 7f2c18096280 0 _get_class not permitted to load kvs -152> 2018-07-25 11:19:42.477 7f2c18096280 0 _get_class not permitted to load lua -151> 2018-07-25 11:19:42.477 7f2c18096280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -150> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -149> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -148> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -147> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -146> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -145> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -144> 2018-07-25 11:19:42.477 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -143> 2018-07-25 11:19:42.517 7f2c18096280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -142> 2018-07-25 11:19:42.517 7f2c18096280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -141> 2018-07-25 11:19:42.517 7f2c18096280 0 _get_class not permitted to load sdk -140> 2018-07-25 11:19:42.517 7f2c18096280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -139> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -138> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -137> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -136> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 load_pgs -135> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 load_pgs opened 0 pgs -134> 2018-07-25 11:19:42.517 7f2c18096280 2 osd.7 0 superblock: I am osd.7 -133> 2018-07-25 11:19:42.517 7f2c18096280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -132> 2018-07-25 11:19:42.517 7f2c18096280 1 Processor -- start -131> 2018-07-25 11:19:42.537 7f2c18096280 1 Processor -- start -130> 2018-07-25 11:19:42.537 7f2c18096280 1 Processor -- start -129> 2018-07-25 11:19:42.537 7f2c18096280 1 Processor -- start -128> 2018-07-25 11:19:42.537 7f2c18096280 1 Processor -- start -127> 2018-07-25 11:19:42.547 7f2c18096280 1 Processor -- start -126> 2018-07-25 11:19:42.547 7f2c18096280 1 Processor -- start -125> 2018-07-25 11:19:42.547 7f2c18096280 10 monclient: init -124> 2018-07-25 11:19:42.547 7f2c18096280 5 adding auth protocol: cephx -123> 2018-07-25 11:19:42.547 7f2c18096280 10 monclient: auth_supported 2 method cephx -122> 2018-07-25 11:19:42.547 7f2c18096280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -121> 2018-07-25 11:19:42.547 7f2c18096280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -120> 2018-07-25 11:19:42.547 7f2c18096280 -1 osd.7 0 log_to_monitors {default=true} -119> 2018-07-25 11:19:42.547 7f2c18096280 10 monclient: _reopen_session rank -1 -118> 2018-07-25 11:19:42.547 7f2c18096280 10 monclient(hunting): picked mon.mimir con 0x55bb7baad800 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -117> 2018-07-25 11:19:42.547 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55bb7ba1b900 con 0 -116> 2018-07-25 11:19:42.547 7f2c18096280 10 monclient(hunting): _renew_subs -115> 2018-07-25 11:19:42.547 7f2bed02b700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -114> 2018-07-25 11:19:42.567 7f2c09b28700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -113> 2018-07-25 11:19:42.577 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 1 0x55bb7babc900 mon_map magic: 0 v1 -112> 2018-07-25 11:19:42.577 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 2 0x55bb7ba1b900 auth_reply(proto 2 0 (0) Success) v1 -111> 2018-07-25 11:19:42.577 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55bb7babc900 con 0x55bb7baad800 -110> 2018-07-25 11:19:42.577 7f2bfe499700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -109> 2018-07-25 11:19:42.577 7f2bfe499700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -108> 2018-07-25 11:19:42.577 7f2bfe499700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -107> 2018-07-25 11:19:42.577 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2259854248 0 0) 0x55bb7ba1b900 con 0x55bb7baad800 -106> 2018-07-25 11:19:42.577 7f2bfe499700 10 monclient(hunting): my global_id is 220367 -105> 2018-07-25 11:19:42.577 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55bb7be12000 con 0 -104> 2018-07-25 11:19:42.587 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 3 0x55bb7be12000 auth_reply(proto 2 0 (0) Success) v1 -103> 2018-07-25 11:19:42.587 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1903350231 0 0) 0x55bb7be12000 con 0x55bb7baad800 -102> 2018-07-25 11:19:42.587 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55bb7ba1b900 con 0 -101> 2018-07-25 11:19:42.587 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 4 0x55bb7ba1b900 auth_reply(proto 2 0 (0) Success) v1 -100> 2018-07-25 11:19:42.587 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (408667462 0 0) 0x55bb7ba1b900 con 0x55bb7baad800 -99> 2018-07-25 11:19:42.587 7f2bfe499700 1 monclient: found mon.mimir -98> 2018-07-25 11:19:42.587 7f2bfe499700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -97> 2018-07-25 11:19:42.587 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb7babc240 con 0 -96> 2018-07-25 11:19:42.587 7f2bfe499700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:19:12.596906) -95> 2018-07-25 11:19:42.587 7f2bfe499700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -94> 2018-07-25 11:19:42.587 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55bb7be12000 con 0 -93> 2018-07-25 11:19:42.587 7f2c18096280 5 monclient: authenticate success, global_id 220367 -92> 2018-07-25 11:19:42.587 7f2c18096280 10 monclient: wait_auth_rotating waiting (until 2018-07-25 11:20:12.597179) -91> 2018-07-25 11:19:42.597 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 5 0x55bb7babc240 config(0 keys) v1 -90> 2018-07-25 11:19:42.597 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 6 0x55bb7be10fc0 mon_map magic: 0 v1 -89> 2018-07-25 11:19:42.597 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 7 0x55bb7be12000 auth_reply(proto 2 0 (0) Success) v1 -88> 2018-07-25 11:19:42.597 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55bb7babc240 con 0x55bb7baad800 -87> 2018-07-25 11:19:42.597 7f2bfe499700 10 monclient: handle_config config(0 keys) v1 -86> 2018-07-25 11:19:42.597 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55bb7be10fc0 con 0x55bb7baad800 -85> 2018-07-25 11:19:42.597 7f2bfe499700 10 monclient: handle_monmap mon_map magic: 0 v1 -84> 2018-07-25 11:19:42.597 7f2bfe499700 10 monclient: got monmap 3, mon.mimir is now rank 2 -83> 2018-07-25 11:19:42.597 7f2bfe499700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -82> 2018-07-25 11:19:42.597 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1346616769 0 0) 0x55bb7be12000 con 0x55bb7baad800 -81> 2018-07-25 11:19:42.597 7f2bfe499700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:19:12.604832) -80> 2018-07-25 11:19:42.597 7f2c18096280 10 monclient: wait_auth_rotating done -79> 2018-07-25 11:19:42.597 7f2c18096280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -78> 2018-07-25 11:19:42.597 7f2c18096280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -77> 2018-07-25 11:19:42.597 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x55bb7babd440 con 0 -76> 2018-07-25 11:19:42.597 7f2bf483a700 4 set_mon_vals no callback set -75> 2018-07-25 11:19:42.667 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 8 0x55bb7babd440 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 -74> 2018-07-25 11:19:42.667 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 ==== 207+0+0 (1704255731 0 0) 0x55bb7babd440 con 0x55bb7baad800 -73> 2018-07-25 11:19:42.667 7f2bfe499700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -72> 2018-07-25 11:19:42.667 7f2bfe499700 10 monclient: _finish_command 1 = 0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. -71> 2018-07-25 11:19:42.667 7f2c18096280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -70> 2018-07-25 11:19:42.667 7f2c18096280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -69> 2018-07-25 11:19:42.667 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x55bb7babd680 con 0 -68> 2018-07-25 11:19:42.677 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 9 0x55bb7babd680 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 -67> 2018-07-25 11:19:42.677 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 ==== 245+0+0 (3636428954 0 0) 0x55bb7babd680 con 0x55bb7baad800 -66> 2018-07-25 11:19:42.677 7f2bfe499700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -65> 2018-07-25 11:19:42.677 7f2bfe499700 10 monclient: _finish_command 2 = 0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -64> 2018-07-25 11:19:42.677 7f2c18096280 0 osd.7 0 done with init, starting boot process -63> 2018-07-25 11:19:42.677 7f2c18096280 10 monclient: _renew_subs -62> 2018-07-25 11:19:42.677 7f2c18096280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -61> 2018-07-25 11:19:42.677 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x55bb7b6818c0 con 0 -60> 2018-07-25 11:19:42.677 7f2c18096280 1 osd.7 0 start_boot -59> 2018-07-25 11:19:42.677 7f2c18096280 10 monclient: get_version osdmap req 0x55bb7be0c3e0 -58> 2018-07-25 11:19:42.677 7f2c18096280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -57> 2018-07-25 11:19:42.677 7f2c18096280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x55bb7bba5800 con 0 -56> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command status hook 0x55bb7b64a830 -55> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command flush_journal hook 0x55bb7b64a830 -54> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_ops_in_flight hook 0x55bb7b64a830 -53> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command ops hook 0x55bb7b64a830 -52> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_blocked_ops hook 0x55bb7b64a830 -51> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_historic_ops hook 0x55bb7b64a830 -50> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_historic_slow_ops hook 0x55bb7b64a830 -49> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_historic_ops_by_duration hook 0x55bb7b64a830 -48> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_op_pq_state hook 0x55bb7b64a830 -47> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_blacklist hook 0x55bb7b64a830 -46> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_watchers hook 0x55bb7b64a830 -45> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_reservations hook 0x55bb7b64a830 -44> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command get_latest_osdmap hook 0x55bb7b64a830 -43> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command heap hook 0x55bb7b64a830 -42> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command set_heap_property hook 0x55bb7b64a830 -41> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command get_heap_property hook 0x55bb7b64a830 -40> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_objectstore_kv_stats hook 0x55bb7b64a830 -39> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_scrubs hook 0x55bb7b64a830 -38> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command calc_objectstore_db_histogram hook 0x55bb7b64a830 -37> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command flush_store_cache hook 0x55bb7b64a830 -36> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command dump_pgstate_history hook 0x55bb7b64a830 -35> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command compact hook 0x55bb7b64a830 -34> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command get_mapped_pools hook 0x55bb7b64a830 -33> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command smart hook 0x55bb7b64a830 -32> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command list_devices hook 0x55bb7b64a830 -31> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command setomapval hook 0x55bb7be0ca20 -30> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command rmomapkey hook 0x55bb7be0ca20 -29> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command setomapheader hook 0x55bb7be0ca20 -28> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command getomap hook 0x55bb7be0ca20 -27> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command truncobj hook 0x55bb7be0ca20 -26> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command injectdataerr hook 0x55bb7be0ca20 -25> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command injectmdataerr hook 0x55bb7be0ca20 -24> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command set_recovery_delay hook 0x55bb7be0ca20 -23> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command trigger_scrub hook 0x55bb7be0ca20 -22> 2018-07-25 11:19:42.677 7f2c18096280 5 asok(0x55bb7b680240) register_command injectfull hook 0x55bb7be0ca20 -21> 2018-07-25 11:19:42.677 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 10 0x55bb7bbc9800 mgrmap(e 34) v1 -20> 2018-07-25 11:19:42.677 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 11 0x55bb7baba000 mon_get_version_reply(handle=1 version=2922) v2 -19> 2018-07-25 11:19:42.677 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x55bb7bbc9800 con 0x55bb7baad800 -18> 2018-07-25 11:19:42.677 7f2bfe499700 4 mgrc handle_mgr_map Got map version 34 -17> 2018-07-25 11:19:42.677 7f2bfe499700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -16> 2018-07-25 11:19:42.677 7f2bfe499700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -15> 2018-07-25 11:19:42.677 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x55bb7be34000 con 0 -14> 2018-07-25 11:19:42.697 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2922) v2 ==== 24+0+0 (2719405947 0 0) 0x55bb7baba000 con 0x55bb7baad800 -13> 2018-07-25 11:19:42.697 7f2bfe499700 10 monclient: handle_get_version_reply finishing 0x55bb7be0c3e0 version 2922 -12> 2018-07-25 11:19:42.697 7f2bf483a700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -11> 2018-07-25 11:19:42.697 7f2bf483a700 -1 osd.7 0 waiting for initial osdmap -10> 2018-07-25 11:19:42.697 7f2bf483a700 10 monclient: _renew_subs -9> 2018-07-25 11:19:42.697 7f2bf483a700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -8> 2018-07-25 11:19:42.697 7f2bf483a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x55bb7babc240 con 0 -7> 2018-07-25 11:19:42.697 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 12 0x55bb7be1e2c0 osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 -6> 2018-07-25 11:19:42.697 7f2bfe499700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 ==== 4888+0+0 (2709902744 0 0) 0x55bb7be1e2c0 con 0x55bb7baad800 -5> 2018-07-25 11:19:42.697 7f2bfe499700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2922] -4> 2018-07-25 11:19:42.697 7f2bfe499700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes -3> 2018-07-25 11:19:42.697 7f2c09327700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55bb7be30000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -2> 2018-07-25 11:19:42.697 7f2c09b28700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55bb7baad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62457 cs=1 l=1). rx mon.2 seq 13 0x55bb7be1e580 osd_map(2320..2359 src has 2319..2922) v4 -1> 2018-07-25 11:19:42.697 7f2c09327700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1472 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55bb7be30000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339633 cs=1 l=1). rx mgr.34109 seq 1 0x55bb7bd1c000 mgrconfigure(period=5, threshold=5) v2 0> 2018-07-25 11:19:42.697 7f2bfe499700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f2bfe499700 time 2018-07-25 11:19:42.707508 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f2c0f4feca2] 2: (()+0x283e67) [0x7f2c0f4fee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55bb796a9efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55bb7965dff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55bb79666d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55bb79667066] 7: (DispatchQueue::entry()+0xb92) [0x7f2c0f5774a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2c0f614dad] 9: (()+0x792a) [0x7f2c0da4d92a] 10: (clone()+0x3f) [0x7f2c0c9d887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:19:42.737 7f2bfe499700 -1 *** Caught signal (Aborted) ** in thread 7f2bfe499700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55bb79b79120] 2: (()+0x14120) [0x7f2c0da5a120] 3: (gsignal()+0x110) [0x7f2c0c90bfc0] 4: (abort()+0x1c7) [0x7f2c0c90db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f2c0f4fedf0] 6: (()+0x283e67) [0x7f2c0f4fee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55bb796a9efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55bb7965dff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55bb79666d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55bb79667066] 11: (DispatchQueue::entry()+0xb92) [0x7f2c0f5774a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2c0f614dad] 13: (()+0x792a) [0x7f2c0da4d92a] 14: (clone()+0x3f) [0x7f2c0c9d887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:19:42.737 7f2bfe499700 -1 *** Caught signal (Aborted) ** in thread 7f2bfe499700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55bb79b79120] 2: (()+0x14120) [0x7f2c0da5a120] 3: (gsignal()+0x110) [0x7f2c0c90bfc0] 4: (abort()+0x1c7) [0x7f2c0c90db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f2c0f4fedf0] 6: (()+0x283e67) [0x7f2c0f4fee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55bb796a9efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55bb7965dff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55bb79666d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55bb79667066] 11: (DispatchQueue::entry()+0xb92) [0x7f2c0f5774a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2c0f614dad] 13: (()+0x792a) [0x7f2c0da4d92a] 14: (clone()+0x3f) [0x7f2c0c9d887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:20:08.858 7fb8296a6280 0 set uid:gid to 114:993 (ceph:ceph) 2018-07-25 11:20:08.858 7fb8296a6280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1563 2018-07-25 11:20:08.858 7fb8296a6280 0 pidfile_write: ignore empty --pid-file 2018-07-25 11:20:08.858 7fb8296a6280 -1 asok(0x55e181d98240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory 2018-07-25 11:20:08.938 7fb8296a6280 0 load: jerasure load: lrc load: isa 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:20:08.978 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:20:08.978 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) close 2018-07-25 11:20:09.298 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:20:09.298 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block 2018-07-25 11:20:09.308 7fb8296a6280 1 bdev(0x55e182296380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational 2018-07-25 11:20:09.308 7fb8296a6280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB 2018-07-25 11:20:09.308 7fb8296a6280 1 bluefs mount 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compression = kNoCompression 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option max_write_buffer_number = 4 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option recycle_log_file_num = 4 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option write_buffer_size = 268435456 2018-07-25 11:20:09.468 7fb8296a6280 1 rocksdb: do_open column families: [default] 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: RocksDB version: 5.13.0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Compile date Jul 24 2018 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: DB SUMMARY 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: CURRENT file: CURRENT 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: IDENTITY file: IDENTITY 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: MANIFEST file: MANIFEST-000029 size: 168 Bytes 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000024.sst 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Write Ahead Log file in db: 000030.log size: 0 ; 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.error_if_exists: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.create_if_missing: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.paranoid_checks: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.env: 0x55e1820bd3e0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.info_log: 0x55e1820df660 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.statistics: (nil) 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_fsync: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_log_file_size: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_fallocate: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_direct_reads: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.create_missing_column_families: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.db_log_dir: 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_dir: db 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_subcompactions: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_flushes: -1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.advise_random_on_open: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_buffer_manager: 0x55e1820dbc20 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.rate_limiter: (nil) 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_pipelined_write: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.row_cache: None 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_filter: None 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_ingest_behind: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.preserve_deletes: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.two_write_queues: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.manual_wal_flush: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_jobs: 2 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_compactions: -1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_total_wal_size: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_open_files: -1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bytes_per_sync: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Compression algorithms supported: 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZSTDNotFinalCompression supported: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZSTD supported: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kXpressCompression supported: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kLZ4HCCompression supported: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kLZ4Compression supported: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kBZip2Compression supported: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZlibCompression supported: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kSnappyCompression supported: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Fast CRC32 supported: Not supported on x86 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000029 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_filter: None 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_filter_factory: None 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e181d613c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55e181e13490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression: NoCompression 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.num_levels: 7 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.level: -1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.arena_block_size: 33554432 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_properties_collectors: 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.inplace_update_support: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bloom_locality: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_successive_merges: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.force_consistency_checks: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.ttl: 0 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000029 succeeded,manifest_file_number is 29, next_file_number is 32, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 27 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 28 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535609494524, "job": 1, "event": "recovery_started", "log_files": [30]} 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #30 mode 0 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 32 2018-07-25 11:20:09.508 7fb8296a6280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535609521804, "job": 1, "event": "recovery_finished"} 2018-07-25 11:20:09.558 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55e1821e5000 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-25 11:20:09.558 7fb8296a6280 1 freelist init 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents 2018-07-25 11:20:09.568 7fb8296a6280 0 _get_class not permitted to load kvs 2018-07-25 11:20:09.568 7fb8296a6280 0 _get_class not permitted to load lua 2018-07-25 11:20:09.568 7fb8296a6280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-25 11:20:09.648 7fb8296a6280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-25 11:20:09.648 7fb8296a6280 0 _get_class not permitted to load sdk 2018-07-25 11:20:09.648 7fb8296a6280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 load_pgs 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 load_pgs opened 0 pgs 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. 2018-07-25 11:20:09.688 7fb8296a6280 -1 osd.7 0 log_to_monitors {default=true} 2018-07-25 11:20:10.028 7fb8296a6280 0 osd.7 0 done with init, starting boot process 2018-07-25 11:20:10.028 7fb8296a6280 1 osd.7 0 start_boot 2018-07-25 11:20:10.028 7fb805e4a700 -1 osd.7 0 waiting for initial osdmap 2018-07-25 11:20:10.058 7fb80faa9700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes 2018-07-25 11:20:10.058 7fb80faa9700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fb80faa9700 time 2018-07-25 11:20:10.064941 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fb820b0eca2] 2: (()+0x283e67) [0x7fb820b0ee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55e1808d5efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55e180889ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55e180892d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55e180893066] 7: (DispatchQueue::entry()+0xb92) [0x7fb820b874a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fb820c24dad] 9: (()+0x792a) [0x7fb81f05d92a] 10: (clone()+0x3f) [0x7fb81dfe887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -449> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perfcounters_dump hook 0x55e181d62060 -448> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command 1 hook 0x55e181d62060 -447> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perf dump hook 0x55e181d62060 -446> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perfcounters_schema hook 0x55e181d62060 -445> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perf histogram dump hook 0x55e181d62060 -444> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command 2 hook 0x55e181d62060 -443> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perf schema hook 0x55e181d62060 -442> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perf histogram schema hook 0x55e181d62060 -441> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command perf reset hook 0x55e181d62060 -440> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config show hook 0x55e181d62060 -439> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config help hook 0x55e181d62060 -438> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config set hook 0x55e181d62060 -437> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config unset hook 0x55e181d62060 -436> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config get hook 0x55e181d62060 -435> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config diff hook 0x55e181d62060 -434> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command config diff get hook 0x55e181d62060 -433> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command log flush hook 0x55e181d62060 -432> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command log dump hook 0x55e181d62060 -431> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command log reopen hook 0x55e181d62060 -430> 2018-07-25 11:20:08.728 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_mempools hook 0x55e1820e4068 -429> 2018-07-25 11:20:08.758 7fb8296a6280 10 monclient: get_monmap_and_config -428> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient: build_initial_monmap -427> 2018-07-25 11:20:08.788 7fb81b939700 2 Event(0x55e181d99040 nevent=5000 time_id=1).set_owner idx=0 owner=140428713367296 -426> 2018-07-25 11:20:08.788 7fb81a937700 2 Event(0x55e181d994c0 nevent=5000 time_id=1).set_owner idx=2 owner=140428696581888 -425> 2018-07-25 11:20:08.788 7fb81b138700 2 Event(0x55e181d99280 nevent=5000 time_id=1).set_owner idx=1 owner=140428704974592 -424> 2018-07-25 11:20:08.788 7fb8296a6280 1 Processor -- start -423> 2018-07-25 11:20:08.788 7fb8296a6280 1 -- - start start -422> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient: init -421> 2018-07-25 11:20:08.788 7fb8296a6280 5 adding auth protocol: cephx -420> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient: auth_supported 2 method cephx -419> 2018-07-25 11:20:08.788 7fb8296a6280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -418> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient: _reopen_session rank -1 -417> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient(hunting): picked mon.mimir con 0x55e1821c2700 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -416> 2018-07-25 11:20:08.788 7fb8296a6280 1 -- - --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55e182132780 con 0 -415> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient(hunting): _renew_subs -414> 2018-07-25 11:20:08.788 7fb8296a6280 10 monclient(hunting): authenticate will time out at 2018-07-25 11:25:08.798483 -413> 2018-07-25 11:20:08.808 7fb81b138700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 -412> 2018-07-25 11:20:08.808 7fb81b138700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -411> 2018-07-25 11:20:08.808 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 1 0x55e1821d4000 mon_map magic: 0 v1 -410> 2018-07-25 11:20:08.808 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 2 0x55e182132780 auth_reply(proto 2 0 (0) Success) v1 -409> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55e1821d4000 con 0x55e1821c2700 -408> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -407> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -406> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -405> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2741913177 0 0) 0x55e182132780 con 0x55e1821c2700 -404> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient(hunting): my global_id is 220370 -403> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55e182132f00 con 0 -402> 2018-07-25 11:20:08.808 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 3 0x55e182132f00 auth_reply(proto 2 0 (0) Success) v1 -401> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2913082136 0 0) 0x55e182132f00 con 0x55e1821c2700 -400> 2018-07-25 11:20:08.808 7fb81a136700 1 monclient: found mon.mimir -399> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -398> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e181d99680 con 0 -397> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:19:38.819114) -396> 2018-07-25 11:20:08.808 7fb81a136700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -395> 2018-07-25 11:20:08.808 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55e182132780 con 0 -394> 2018-07-25 11:20:08.808 7fb8296a6280 5 monclient: authenticate success, global_id 220370 -393> 2018-07-25 11:20:08.818 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 4 0x55e181d99680 config(0 keys) v1 -392> 2018-07-25 11:20:08.818 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 5 0x55e1821d4480 mon_map magic: 0 v1 -391> 2018-07-25 11:20:08.818 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62458 cs=1 l=1). rx mon.2 seq 6 0x55e182132780 auth_reply(proto 2 0 (0) Success) v1 -390> 2018-07-25 11:20:08.818 7fb81a136700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55e181d99680 con 0x55e1821c2700 -389> 2018-07-25 11:20:08.818 7fb81a136700 10 monclient: handle_config config(0 keys) v1 -388> 2018-07-25 11:20:08.818 7fb818933700 4 set_mon_vals no callback set -387> 2018-07-25 11:20:08.818 7fb8296a6280 10 monclient: get_monmap_and_config success -386> 2018-07-25 11:20:08.818 7fb8296a6280 10 monclient: shutdown -385> 2018-07-25 11:20:08.818 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN pgs=62458 cs=1 l=1).mark_down -384> 2018-07-25 11:20:08.818 7fb8296a6280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c2700 :-1 s=STATE_OPEN pgs=62458 cs=1 l=1)._stop -383> 2018-07-25 11:20:08.838 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 shutdown_connections -382> 2018-07-25 11:20:08.838 7fb8296a6280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 shutdown_connections mark down [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 0x55e1821c2700 -381> 2018-07-25 11:20:08.838 7fb8296a6280 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 shutdown_connections delete 0x55e1821c2700 -380> 2018-07-25 11:20:08.848 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 shutdown_connections -379> 2018-07-25 11:20:08.848 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 wait complete. -378> 2018-07-25 11:20:08.848 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 conn(0x55e1821c2000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -377> 2018-07-25 11:20:08.848 7fb8296a6280 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 >> [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/2373921861 conn(0x55e1821c2000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -376> 2018-07-25 11:20:08.858 7fb8296a6280 0 set uid:gid to 114:993 (ceph:ceph) -375> 2018-07-25 11:20:08.858 7fb8296a6280 0 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable), process (unknown), pid 1563 -374> 2018-07-25 11:20:08.858 7fb8296a6280 0 pidfile_write: ignore empty --pid-file -373> 2018-07-25 11:20:08.858 7fb8296a6280 5 asok(0x55e181d98240) init /var/run/ceph/ceph-osd.7.asok -372> 2018-07-25 11:20:08.858 7fb8296a6280 5 asok(0x55e181d98240) bind_and_listen /var/run/ceph/ceph-osd.7.asok -371> 2018-07-25 11:20:08.858 7fb8296a6280 -1 asok(0x55e181d98240) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-osd.7.asok': (2) No such file or directory -370> 2018-07-25 11:20:08.858 7fb8296a6280 5 object store type is bluestore -369> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -368> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 -367> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -366> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1563 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1563 -365> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 learned_addr learned my addr [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:0/0 -364> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1563 _finish_bind bind my_inst.addr is [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1563 -363> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 learned_addr learned my addr [fda8:941:2491:1699:b45:a2e6:1383:2b98]:0/0 -362> 2018-07-25 11:20:08.918 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1563 _finish_bind bind my_inst.addr is [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1563 -361> 2018-07-25 11:20:08.918 7fb8296a6280 10 monclient: build_initial_monmap -360> 2018-07-25 11:20:08.938 7fb8296a6280 0 load: jerasure load: lrc load: isa -359> 2018-07-25 11:20:08.968 7fb8296a6280 5 adding auth protocol: cephx -358> 2018-07-25 11:20:08.968 7fb8296a6280 5 adding auth protocol: cephx -357> 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -356> 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -355> 2018-07-25 11:20:08.968 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -354> 2018-07-25 11:20:08.978 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -353> 2018-07-25 11:20:08.978 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) close -352> 2018-07-25 11:20:09.298 7fb8296a6280 5 asok(0x55e181d98240) register_command objecter_requests hook 0x55e181d62650 -351> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 start start -350> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- - start start -349> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- - start start -348> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6802/1563 start start -347> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6801/1563 start start -346> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- [fdd7:d94b:3c2e:b69f:b7c6:ef9:7f21:2a96]:6800/1563 start start -345> 2018-07-25 11:20:09.298 7fb8296a6280 1 -- - start start -344> 2018-07-25 11:20:09.298 7fb8296a6280 2 osd.7 0 init /var/lib/ceph/osd/ceph-7 (looks like hdd) -343> 2018-07-25 11:20:09.298 7fb8296a6280 2 osd.7 0 journal /var/lib/ceph/osd/ceph-7/journal -342> 2018-07-25 11:20:09.298 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _mount path /var/lib/ceph/osd/ceph-7 -341> 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -340> 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -339> 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296000 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -338> 2018-07-25 11:20:09.298 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 -337> 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev create path /var/lib/ceph/osd/ceph-7/block type kernel -336> 2018-07-25 11:20:09.298 7fb8296a6280 1 bdev(0x55e182296380 /var/lib/ceph/osd/ceph-7/block) open path /var/lib/ceph/osd/ceph-7/block -335> 2018-07-25 11:20:09.308 7fb8296a6280 1 bdev(0x55e182296380 /var/lib/ceph/osd/ceph-7/block) open size 159995920384 (0x2540800000, 149 GiB) block_size 4096 (4 KiB) rotational -334> 2018-07-25 11:20:09.308 7fb8296a6280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-7/block size 149 GiB -333> 2018-07-25 11:20:09.308 7fb8296a6280 1 bluefs mount -332> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compaction_readahead_size = 2097152 -331> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compression = kNoCompression -330> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option max_write_buffer_number = 4 -329> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -328> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option recycle_log_file_num = 4 -327> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option writable_file_max_buffer_size = 0 -326> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option write_buffer_size = 268435456 -325> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compaction_readahead_size = 2097152 -324> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option compression = kNoCompression -323> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option max_write_buffer_number = 4 -322> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option min_write_buffer_number_to_merge = 1 -321> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option recycle_log_file_num = 4 -320> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option writable_file_max_buffer_size = 0 -319> 2018-07-25 11:20:09.448 7fb8296a6280 0 set rocksdb option write_buffer_size = 268435456 -318> 2018-07-25 11:20:09.468 7fb8296a6280 1 rocksdb: do_open column families: [default] -317> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: RocksDB version: 5.13.0 -316> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -315> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Compile date Jul 24 2018 -314> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: DB SUMMARY -313> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: CURRENT file: CURRENT -312> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: IDENTITY file: IDENTITY -311> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: MANIFEST file: MANIFEST-000029 size: 168 Bytes -310> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: SST files in db dir, Total Num: 1, files: 000024.sst -309> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Write Ahead Log file in db: 000030.log size: 0 ; -308> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.error_if_exists: 0 -307> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.create_if_missing: 0 -306> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.paranoid_checks: 1 -305> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.env: 0x55e1820bd3e0 -304> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.info_log: 0x55e1820df660 -303> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_file_opening_threads: 16 -302> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.statistics: (nil) -301> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_fsync: 0 -300> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_log_file_size: 0 -299> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -298> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.log_file_time_to_roll: 0 -297> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.keep_log_file_num: 1000 -296> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.recycle_log_file_num: 4 -295> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_fallocate: 1 -294> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_mmap_reads: 0 -293> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_mmap_writes: 0 -292> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_direct_reads: 0 -291> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -290> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.create_missing_column_families: 0 -289> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.db_log_dir: -288> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_dir: db -287> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_cache_numshardbits: 6 -286> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_subcompactions: 1 -285> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_flushes: -1 -284> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.WAL_ttl_seconds: 0 -283> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.WAL_size_limit_MB: 0 -282> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.manifest_preallocation_size: 4194304 -281> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.is_fd_close_on_exec: 1 -280> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.advise_random_on_open: 1 -279> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.db_write_buffer_size: 0 -278> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_buffer_manager: 0x55e1820dbc20 -277> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.access_hint_on_compaction_start: 1 -276> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -275> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -274> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.use_adaptive_mutex: 0 -273> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.rate_limiter: (nil) -272> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -271> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_recovery_mode: 0 -270> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_thread_tracking: 0 -269> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_pipelined_write: 0 -268> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -267> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -266> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_thread_max_yield_usec: 100 -265> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -264> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.row_cache: None -263> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_filter: None -262> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.avoid_flush_during_recovery: 0 -261> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.allow_ingest_behind: 0 -260> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.preserve_deletes: 0 -259> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.two_write_queues: 0 -258> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.manual_wal_flush: 0 -257> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_jobs: 2 -256> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_background_compactions: -1 -255> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -254> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.writable_file_max_buffer_size: 0 -253> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.delayed_write_rate : 16777216 -252> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_total_wal_size: 0 -251> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -250> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.stats_dump_period_sec: 600 -249> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_open_files: -1 -248> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bytes_per_sync: 0 -247> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.wal_bytes_per_sync: 0 -246> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_readahead_size: 2097152 -245> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Compression algorithms supported: -244> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZSTDNotFinalCompression supported: 0 -243> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZSTD supported: 0 -242> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kXpressCompression supported: 0 -241> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kLZ4HCCompression supported: 1 -240> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kLZ4Compression supported: 1 -239> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kBZip2Compression supported: 0 -238> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kZlibCompression supported: 1 -237> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: kSnappyCompression supported: 1 -236> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Fast CRC32 supported: Not supported on x86 -235> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000029 -234> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]: -233> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -232> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -231> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_filter: None -230> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_filter_factory: None -229> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_factory: SkipListFactory -228> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_factory: BlockBasedTable -227> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e181d613c8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55e181e13490 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 -226> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.write_buffer_size: 268435456 -225> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_write_buffer_number: 4 -224> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression: NoCompression -223> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bottommost_compression: Disabled -222> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.prefix_extractor: nullptr -221> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -220> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.num_levels: 7 -219> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -218> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -217> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.window_bits: -14 -216> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.level: -1 -215> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.strategy: 0 -214> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -213> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 -212> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -211> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -210> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level0_stop_writes_trigger: 36 -209> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.target_file_size_base: 67108864 -208> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.target_file_size_multiplier: 1 -207> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -206> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -205> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -204> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -203> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -202> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -201> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -200> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -199> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -198> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -197> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -196> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_compaction_bytes: 1677721600 -195> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.arena_block_size: 33554432 -194> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -193> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -192> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -191> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.disable_auto_compactions: 0 -190> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -189> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_pri: kByCompensatedSize -188> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -187> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -186> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -185> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -184> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -183> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize -182> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -181> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 -180> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.compaction_options_fifo.ttl: 0 -179> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.table_properties_collectors: -178> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.inplace_update_support: 0 -177> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.inplace_update_num_locks: 10000 -176> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -175> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.memtable_huge_page_size: 0 -174> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.bloom_locality: 0 -173> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.max_successive_merges: 0 -172> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.optimize_filters_for_hits: 0 -171> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.paranoid_file_checks: 0 -170> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.force_consistency_checks: 0 -169> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.report_bg_io_stats: 0 -168> 2018-07-25 11:20:09.468 7fb8296a6280 4 rocksdb: Options.ttl: 0 -167> 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000029 succeeded,manifest_file_number is 29, next_file_number is 32, last_sequence is 29, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 27 -166> 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 28 -165> 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535609494524, "job": 1, "event": "recovery_started", "log_files": [30]} -164> 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:551] Recovering log #30 mode 0 -163> 2018-07-25 11:20:09.488 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/version_set.cc:2863] Creating manifest 32 -162> 2018-07-25 11:20:09.508 7fb8296a6280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532535609521804, "job": 1, "event": "recovery_finished"} -161> 2018-07-25 11:20:09.508 7fb8296a6280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-000029 type=3 #29 -- OK -160> 2018-07-25 11:20:09.528 7fb8296a6280 5 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//000030.log type=0 #30 -- OK -159> 2018-07-25 11:20:09.558 7fb8296a6280 4 rocksdb: [/var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55e1821e5000 -158> 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -157> 2018-07-25 11:20:09.558 7fb8296a6280 1 freelist init -156> 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc opening allocation metadata -155> 2018-07-25 11:20:09.558 7fb8296a6280 1 bluestore(/var/lib/ceph/osd/ceph-7) _open_alloc loaded 149 GiB in 1 extents -154> 2018-07-25 11:20:09.558 7fb8296a6280 2 osd.7 0 journal looks like hdd -153> 2018-07-25 11:20:09.558 7fb8296a6280 2 osd.7 0 boot -152> 2018-07-25 11:20:09.568 7fb8296a6280 0 _get_class not permitted to load kvs -151> 2018-07-25 11:20:09.568 7fb8296a6280 0 _get_class not permitted to load lua -150> 2018-07-25 11:20:09.568 7fb8296a6280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -149> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/rgw/cls_rgw.cc:3833: Loaded rgw class! -148> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/user/cls_user.cc:410: Loaded user class! -147> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -146> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -145> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -144> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/log/cls_log.cc:299: Loaded log class! -143> 2018-07-25 11:20:09.568 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/version/cls_version.cc:214: Loaded version class! -142> 2018-07-25 11:20:09.648 7fb8296a6280 1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/refcount/cls_refcount.cc:399: Loaded refcount class! -141> 2018-07-25 11:20:09.648 7fb8296a6280 0 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello -140> 2018-07-25 11:20:09.648 7fb8296a6280 0 _get_class not permitted to load sdk -139> 2018-07-25 11:20:09.648 7fb8296a6280 1 osd.7 0 warning: got an error loading one or more classes: (1) Operation not permitted -138> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for clients -137> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons -136> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 crush map has features 288232575208783872, adjusting msgr requires for osds -135> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 load_pgs -134> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 load_pgs opened 0 pgs -133> 2018-07-25 11:20:09.648 7fb8296a6280 2 osd.7 0 superblock: I am osd.7 -132> 2018-07-25 11:20:09.648 7fb8296a6280 0 osd.7 0 using weightedpriority op queue with priority op cut off at 64. -131> 2018-07-25 11:20:09.648 7fb8296a6280 1 Processor -- start -130> 2018-07-25 11:20:09.678 7fb8296a6280 1 Processor -- start -129> 2018-07-25 11:20:09.678 7fb8296a6280 1 Processor -- start -128> 2018-07-25 11:20:09.678 7fb8296a6280 1 Processor -- start -127> 2018-07-25 11:20:09.678 7fb8296a6280 1 Processor -- start -126> 2018-07-25 11:20:09.678 7fb8296a6280 1 Processor -- start -125> 2018-07-25 11:20:09.688 7fb8296a6280 1 Processor -- start -124> 2018-07-25 11:20:09.688 7fb8296a6280 10 monclient: init -123> 2018-07-25 11:20:09.688 7fb8296a6280 5 adding auth protocol: cephx -122> 2018-07-25 11:20:09.688 7fb8296a6280 10 monclient: auth_supported 2 method cephx -121> 2018-07-25 11:20:09.688 7fb8296a6280 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring -120> 2018-07-25 11:20:09.688 7fb8296a6280 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -119> 2018-07-25 11:20:09.688 7fb8296a6280 -1 osd.7 0 log_to_monitors {default=true} -118> 2018-07-25 11:20:09.718 7fb8296a6280 10 monclient: _reopen_session rank -1 -117> 2018-07-25 11:20:09.718 7fb8296a6280 10 monclient(hunting): picked mon.mimir con 0x55e1821c5800 addr [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -116> 2018-07-25 11:20:09.718 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55e182132780 con 0 -115> 2018-07-25 11:20:09.718 7fb8296a6280 10 monclient(hunting): _renew_subs -114> 2018-07-25 11:20:09.718 7fb7fe63b700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -113> 2018-07-25 11:20:09.748 7fb81b138700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -112> 2018-07-25 11:20:09.758 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 1 0x55e1821d4900 mon_map magic: 0 v1 -111> 2018-07-25 11:20:09.758 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 2 0x55e182132780 auth_reply(proto 2 0 (0) Success) v1 -110> 2018-07-25 11:20:09.758 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 1 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55e1821d4900 con 0x55e1821c5800 -109> 2018-07-25 11:20:09.758 7fb80faa9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -108> 2018-07-25 11:20:09.758 7fb80faa9700 10 monclient(hunting): got monmap 3, mon.mimir is now rank 2 -107> 2018-07-25 11:20:09.758 7fb80faa9700 10 monclient(hunting): dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -106> 2018-07-25 11:20:09.758 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (4131017418 0 0) 0x55e182132780 con 0x55e1821c5800 -105> 2018-07-25 11:20:09.758 7fb80faa9700 10 monclient(hunting): my global_id is 220373 -104> 2018-07-25 11:20:09.758 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55e182133680 con 0 -103> 2018-07-25 11:20:09.768 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 3 0x55e182133680 auth_reply(proto 2 0 (0) Success) v1 -102> 2018-07-25 11:20:09.778 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2813837725 0 0) 0x55e182133680 con 0x55e1821c5800 -101> 2018-07-25 11:20:09.778 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55e182132780 con 0 -100> 2018-07-25 11:20:09.788 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 4 0x55e182132780 auth_reply(proto 2 0 (0) Success) v1 -99> 2018-07-25 11:20:09.788 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (3851815083 0 0) 0x55e182132780 con 0x55e1821c5800 -98> 2018-07-25 11:20:09.788 7fb80faa9700 1 monclient: found mon.mimir -97> 2018-07-25 11:20:09.788 7fb80faa9700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -96> 2018-07-25 11:20:09.788 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1821d4240 con 0 -95> 2018-07-25 11:20:09.788 7fb80faa9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-07-25 11:19:39.794642) -94> 2018-07-25 11:20:09.788 7fb80faa9700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -93> 2018-07-25 11:20:09.788 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55e182133680 con 0 -92> 2018-07-25 11:20:09.788 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 5 0x55e1821d4240 config(0 keys) v1 -91> 2018-07-25 11:20:09.788 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 6 0x55e182532000 mon_map magic: 0 v1 -90> 2018-07-25 11:20:09.788 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 7 0x55e182133680 auth_reply(proto 2 0 (0) Success) v1 -89> 2018-07-25 11:20:09.788 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55e1821d4240 con 0x55e1821c5800 -88> 2018-07-25 11:20:09.788 7fb80faa9700 10 monclient: handle_config config(0 keys) v1 -87> 2018-07-25 11:20:09.788 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 6 ==== mon_map magic: 0 v1 ==== 475+0+0 (2018016376 0 0) 0x55e182532000 con 0x55e1821c5800 -86> 2018-07-25 11:20:09.788 7fb80faa9700 10 monclient: handle_monmap mon_map magic: 0 v1 -85> 2018-07-25 11:20:09.798 7fb80faa9700 10 monclient: got monmap 3, mon.mimir is now rank 2 -84> 2018-07-25 11:20:09.798 7fb80faa9700 10 monclient: dump: epoch 3 fsid 922292e4-e02a-42e3-b581-34472d56c92c last_changed 2018-07-15 00:27:09.482983 created 2018-07-14 15:50:06.570111 0: [fda8:941:2491:1699:b3b:5c9b:ccb6:3024]:6789/0 mon.hoenir 1: [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6789/0 mon.fenrir 2: [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 mon.mimir -83> 2018-07-25 11:20:09.798 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1544553474 0 0) 0x55e182133680 con 0x55e1821c5800 -82> 2018-07-25 11:20:09.798 7fb80faa9700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-25 11:19:39.804219) -81> 2018-07-25 11:20:09.798 7fb805e4a700 4 set_mon_vals no callback set -80> 2018-07-25 11:20:09.798 7fb8296a6280 5 monclient: authenticate success, global_id 220373 -79> 2018-07-25 11:20:09.798 7fb8296a6280 10 monclient: wait_auth_rotating done -78> 2018-07-25 11:20:09.798 7fb8296a6280 10 monclient: _send_command 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -77> 2018-07-25 11:20:09.798 7fb8296a6280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -76> 2018-07-25 11:20:09.798 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1 -- 0x55e1821d4b40 con 0 -75> 2018-07-25 11:20:09.998 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 8 0x55e1821d4b40 mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 -74> 2018-07-25 11:20:09.998 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]=0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. v2922) v1 ==== 207+0+0 (1704255731 0 0) 0x55e1821d4b40 con 0x55e1821c5800 -73> 2018-07-25 11:20:09.998 7fb80faa9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}] -72> 2018-07-25 11:20:09.998 7fb80faa9700 10 monclient: _finish_command 1 = 0 osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. -71> 2018-07-25 11:20:09.998 7fb8296a6280 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -70> 2018-07-25 11:20:09.998 7fb8296a6280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -69> 2018-07-25 11:20:09.998 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]} v 0) v1 -- 0x55e1821d4d80 con 0 -68> 2018-07-25 11:20:10.028 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 9 0x55e1821d4d80 mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 -67> 2018-07-25 11:20:10.028 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}]=0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map v2922) v1 ==== 245+0+0 (3636428954 0 0) 0x55e1821d4d80 con 0x55e1821c5800 -66> 2018-07-25 11:20:10.028 7fb80faa9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight":0.1455, "args": ["host=fenrir", "root=default"]}] -65> 2018-07-25 11:20:10.028 7fb80faa9700 10 monclient: _finish_command 2 = 0 create-or-move updated item name 'osd.7' weight 0.1455 at location {host=fenrir,root=default} to crush map -64> 2018-07-25 11:20:10.028 7fb8296a6280 0 osd.7 0 done with init, starting boot process -63> 2018-07-25 11:20:10.028 7fb8296a6280 10 monclient: _renew_subs -62> 2018-07-25 11:20:10.028 7fb8296a6280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -61> 2018-07-25 11:20:10.028 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 -- 0x55e181d998c0 con 0 -60> 2018-07-25 11:20:10.028 7fb8296a6280 1 osd.7 0 start_boot -59> 2018-07-25 11:20:10.028 7fb8296a6280 10 monclient: get_version osdmap req 0x55e18251a680 -58> 2018-07-25 11:20:10.028 7fb8296a6280 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -57> 2018-07-25 11:20:10.028 7fb8296a6280 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x55e1822bd200 con 0 -56> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command status hook 0x55e181d62840 -55> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command flush_journal hook 0x55e181d62840 -54> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_ops_in_flight hook 0x55e181d62840 -53> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command ops hook 0x55e181d62840 -52> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_blocked_ops hook 0x55e181d62840 -51> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_historic_ops hook 0x55e181d62840 -50> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_historic_slow_ops hook 0x55e181d62840 -49> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_historic_ops_by_duration hook 0x55e181d62840 -48> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_op_pq_state hook 0x55e181d62840 -47> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_blacklist hook 0x55e181d62840 -46> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_watchers hook 0x55e181d62840 -45> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_reservations hook 0x55e181d62840 -44> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command get_latest_osdmap hook 0x55e181d62840 -43> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command heap hook 0x55e181d62840 -42> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command set_heap_property hook 0x55e181d62840 -41> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command get_heap_property hook 0x55e181d62840 -40> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_objectstore_kv_stats hook 0x55e181d62840 -39> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_scrubs hook 0x55e181d62840 -38> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command calc_objectstore_db_histogram hook 0x55e181d62840 -37> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command flush_store_cache hook 0x55e181d62840 -36> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command dump_pgstate_history hook 0x55e181d62840 -35> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command compact hook 0x55e181d62840 -34> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command get_mapped_pools hook 0x55e181d62840 -33> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command smart hook 0x55e181d62840 -32> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command list_devices hook 0x55e181d62840 -31> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command setomapval hook 0x55e18251aa60 -30> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command rmomapkey hook 0x55e18251aa60 -29> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command setomapheader hook 0x55e18251aa60 -28> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command getomap hook 0x55e18251aa60 -27> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command truncobj hook 0x55e18251aa60 -26> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command injectdataerr hook 0x55e18251aa60 -25> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command injectmdataerr hook 0x55e18251aa60 -24> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command set_recovery_delay hook 0x55e18251aa60 -23> 2018-07-25 11:20:10.028 7fb8296a6280 5 asok(0x55e181d98240) register_command trigger_scrub hook 0x55e18251aa60 -22> 2018-07-25 11:20:10.028 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 10 0x55e1824fb800 mgrmap(e 34) v1 -21> 2018-07-25 11:20:10.028 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 10 ==== mgrmap(e 34) v1 ==== 1856+0+0 (3131008244 0 0) 0x55e1824fb800 con 0x55e1821c5800 -20> 2018-07-25 11:20:10.028 7fb80faa9700 4 mgrc handle_mgr_map Got map version 34 -19> 2018-07-25 11:20:10.028 7fb80faa9700 4 mgrc handle_mgr_map Active mgr is now [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -18> 2018-07-25 11:20:10.028 7fb80faa9700 4 mgrc reconnect Starting new session with [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -17> 2018-07-25 11:20:10.028 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 -- mgropen(unknown.7) v3 -- 0x55e182296a80 con 0 -16> 2018-07-25 11:20:10.028 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 11 0x55e181d71400 mon_get_version_reply(handle=1 version=2922) v2 -15> 2018-07-25 11:20:10.028 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 11 ==== mon_get_version_reply(handle=1 version=2922) v2 ==== 24+0+0 (2719405947 0 0) 0x55e181d71400 con 0x55e1821c5800 -14> 2018-07-25 11:20:10.028 7fb80faa9700 10 monclient: handle_get_version_reply finishing 0x55e18251a680 version 2922 -13> 2018-07-25 11:20:10.028 7fb805e4a700 5 osd.7 0 heartbeat: osd_stat(1.0 GiB used, 148 GiB avail, 149 GiB total, peers [] op hist []) -12> 2018-07-25 11:20:10.028 7fb805e4a700 -1 osd.7 0 waiting for initial osdmap -11> 2018-07-25 11:20:10.048 7fb805e4a700 10 monclient: _renew_subs -10> 2018-07-25 11:20:10.048 7fb805e4a700 10 monclient: _send_mon_message to mon.mimir at [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -9> 2018-07-25 11:20:10.048 7fb805e4a700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 --> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 -- mon_subscribe({osdmap=2318}) v3 -- 0x55e1821d4240 con 0 -8> 2018-07-25 11:20:10.048 7fb8296a6280 5 asok(0x55e181d98240) register_command injectfull hook 0x55e18251aa60 -7> 2018-07-25 11:20:10.048 7fb81a937700 2 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55e182544000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 -6> 2018-07-25 11:20:10.048 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 12 0x55e18240e000 osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 -5> 2018-07-25 11:20:10.048 7fb81b138700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 conn(0x55e1821c5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=62459 cs=1 l=1). rx mon.2 seq 13 0x55e18240e2c0 osd_map(2320..2359 src has 2319..2922) v4 -4> 2018-07-25 11:20:10.048 7fb80faa9700 1 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 <== mon.2 [fda8:941:2491:1699:60fa:e622:8345:2162]:6789/0 12 ==== osd_map(2319..2319 src has 2319..2922 +gap_removed_snaps) v4 ==== 4888+0+0 (2709902744 0 0) 0x55e18240e000 con 0x55e1821c5800 -3> 2018-07-25 11:20:10.048 7fb80faa9700 3 osd.7 0 handle_osd_map epochs [2319,2319], i have 0, src has [2319,2922] -2> 2018-07-25 11:20:10.058 7fb80faa9700 -1 osd.7 0 failed to load OSD map for epoch 2318, got 0 bytes -1> 2018-07-25 11:20:10.058 7fb81a937700 5 -- [fda8:941:2491:1699:b45:a2e6:1383:2b98]:6801/1563 >> [fda8:941:2491:1699:60fa:e622:8345:2162]:6801/254932 conn(0x55e182544000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=339634 cs=1 l=1). rx mgr.34109 seq 1 0x55e18242c000 mgrconfigure(period=5, threshold=5) v2 0> 2018-07-25 11:20:10.058 7fb80faa9700 -1 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fb80faa9700 time 2018-07-25 11:20:10.064941 /var/tmp/portage/sys-cluster/ceph-13.2.0/work/ceph-13.2.0/src/osd/OSD.h: 828: FAILED assert(ret) ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fb820b0eca2] 2: (()+0x283e67) [0x7fb820b0ee67] 3: (OSDService::get_map(unsigned int)+0x4a) [0x55e1808d5efa] 4: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55e180889ff1] 5: (OSD::_dispatch(Message*)+0xa1) [0x55e180892d41] 6: (OSD::ms_dispatch(Message*)+0x56) [0x55e180893066] 7: (DispatchQueue::entry()+0xb92) [0x7fb820b874a2] 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fb820c24dad] 9: (()+0x792a) [0x7fb81f05d92a] 10: (clone()+0x3f) [0x7fb81dfe887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events --- 2018-07-25 11:20:10.118 7fb80faa9700 -1 *** Caught signal (Aborted) ** in thread 7fb80faa9700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55e180da5120] 2: (()+0x14120) [0x7fb81f06a120] 3: (gsignal()+0x110) [0x7fb81df1bfc0] 4: (abort()+0x1c7) [0x7fb81df1db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fb820b0edf0] 6: (()+0x283e67) [0x7fb820b0ee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55e1808d5efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55e180889ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55e180892d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55e180893066] 11: (DispatchQueue::entry()+0xb92) [0x7fb820b874a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fb820c24dad] 13: (()+0x792a) [0x7fb81f05d92a] 14: (clone()+0x3f) [0x7fb81dfe887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-25 11:20:10.118 7fb80faa9700 -1 *** Caught signal (Aborted) ** in thread 7fb80faa9700 thread_name:ms_dispatch ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 1: (()+0x8de120) [0x55e180da5120] 2: (()+0x14120) [0x7fb81f06a120] 3: (gsignal()+0x110) [0x7fb81df1bfc0] 4: (abort()+0x1c7) [0x7fb81df1db07] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fb820b0edf0] 6: (()+0x283e67) [0x7fb820b0ee67] 7: (OSDService::get_map(unsigned int)+0x4a) [0x55e1808d5efa] 8: (OSD::handle_osd_map(MOSDMap*)+0xf61) [0x55e180889ff1] 9: (OSD::_dispatch(Message*)+0xa1) [0x55e180892d41] 10: (OSD::ms_dispatch(Message*)+0x56) [0x55e180893066] 11: (DispatchQueue::entry()+0xb92) [0x7fb820b874a2] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fb820c24dad] 13: (()+0x792a) [0x7fb81f05d92a] 14: (clone()+0x3f) [0x7fb81dfe887f] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.7.log --- end dump of recent events ---