Project

General

Profile

Bug #20933 » ceph-mon.ceph-mon-1.log

Denis Zadonskii, 08/31/2017 10:57 AM

 
2017-08-31 10:42:22.620627 7fcf45c8ae40 0 set uid:gid to 167:167 (ceph:ceph)
2017-08-31 10:42:22.620684 7fcf45c8ae40 0 ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), pid 2412
2017-08-31 10:42:22.621200 7fcf45c8ae40 0 pidfile_write: ignore empty --pid-file
2017-08-31 10:42:22.700474 7fcf45c8ae40 0 load: jerasure load: lrc load: isa
2017-08-31 10:42:22.700948 7fcf45c8ae40 0 set rocksdb option compression = kNoCompression
2017-08-31 10:42:22.701036 7fcf45c8ae40 0 set rocksdb option write_buffer_size = 33554432
2017-08-31 10:42:22.701108 7fcf45c8ae40 0 set rocksdb option compression = kNoCompression
2017-08-31 10:42:22.701122 7fcf45c8ae40 0 set rocksdb option write_buffer_size = 33554432
2017-08-31 10:42:22.701462 7fcf45c8ae40 4 rocksdb: RocksDB version: 5.4.0

2017-08-31 10:42:22.701491 7fcf45c8ae40 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2017-08-31 10:42:22.701495 7fcf45c8ae40 4 rocksdb: Compile date Aug 28 2017
2017-08-31 10:42:22.701501 7fcf45c8ae40 4 rocksdb: DB SUMMARY

2017-08-31 10:42:22.701682 7fcf45c8ae40 4 rocksdb: CURRENT file: CURRENT

2017-08-31 10:42:22.701702 7fcf45c8ae40 4 rocksdb: IDENTITY file: IDENTITY

2017-08-31 10:42:22.701713 7fcf45c8ae40 4 rocksdb: MANIFEST file: MANIFEST-235767 size: 293 Bytes

2017-08-31 10:42:22.701719 7fcf45c8ae40 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph-mon-1/store.db dir, Total Num: 4, files: 235681.sst 235682.sst 235683.sst 235685.sst

2017-08-31 10:42:22.701724 7fcf45c8ae40 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph-mon-1/store.db: 235768.log size: 0 ;

2017-08-31 10:42:22.701730 7fcf45c8ae40 4 rocksdb: Options.error_if_exists: 0
2017-08-31 10:42:22.701733 7fcf45c8ae40 4 rocksdb: Options.create_if_missing: 0
2017-08-31 10:42:22.701735 7fcf45c8ae40 4 rocksdb: Options.paranoid_checks: 1
2017-08-31 10:42:22.701737 7fcf45c8ae40 4 rocksdb: Options.env: 0x7fcf4ed8fec0
2017-08-31 10:42:22.701740 7fcf45c8ae40 4 rocksdb: Options.info_log: 0x7fcf5001b520
2017-08-31 10:42:22.701743 7fcf45c8ae40 4 rocksdb: Options.max_open_files: -1
2017-08-31 10:42:22.701745 7fcf45c8ae40 4 rocksdb: Options.max_file_opening_threads: 16
2017-08-31 10:42:22.701747 7fcf45c8ae40 4 rocksdb: Options.use_fsync: 0
2017-08-31 10:42:22.701750 7fcf45c8ae40 4 rocksdb: Options.max_log_file_size: 0
2017-08-31 10:42:22.701752 7fcf45c8ae40 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-08-31 10:42:22.701755 7fcf45c8ae40 4 rocksdb: Options.log_file_time_to_roll: 0
2017-08-31 10:42:22.701757 7fcf45c8ae40 4 rocksdb: Options.keep_log_file_num: 1000
2017-08-31 10:42:22.701760 7fcf45c8ae40 4 rocksdb: Options.recycle_log_file_num: 0
2017-08-31 10:42:22.701763 7fcf45c8ae40 4 rocksdb: Options.allow_fallocate: 1
2017-08-31 10:42:22.701766 7fcf45c8ae40 4 rocksdb: Options.allow_mmap_reads: 0
2017-08-31 10:42:22.701768 7fcf45c8ae40 4 rocksdb: Options.allow_mmap_writes: 0
2017-08-31 10:42:22.701770 7fcf45c8ae40 4 rocksdb: Options.use_direct_reads: 0
2017-08-31 10:42:22.701772 7fcf45c8ae40 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2017-08-31 10:42:22.701775 7fcf45c8ae40 4 rocksdb: Options.create_missing_column_families: 0
2017-08-31 10:42:22.701777 7fcf45c8ae40 4 rocksdb: Options.db_log_dir:
2017-08-31 10:42:22.701780 7fcf45c8ae40 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-ceph-mon-1/store.db
2017-08-31 10:42:22.701828 7fcf45c8ae40 4 rocksdb: Options.table_cache_numshardbits: 6
2017-08-31 10:42:22.701832 7fcf45c8ae40 4 rocksdb: Options.max_subcompactions: 1
2017-08-31 10:42:22.701835 7fcf45c8ae40 4 rocksdb: Options.max_background_flushes: 1
2017-08-31 10:42:22.701837 7fcf45c8ae40 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-08-31 10:42:22.701840 7fcf45c8ae40 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-08-31 10:42:22.701842 7fcf45c8ae40 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-08-31 10:42:22.701844 7fcf45c8ae40 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-08-31 10:42:22.701846 7fcf45c8ae40 4 rocksdb: Options.advise_random_on_open: 1
2017-08-31 10:42:22.701848 7fcf45c8ae40 4 rocksdb: Options.db_write_buffer_size: 0
2017-08-31 10:42:22.701851 7fcf45c8ae40 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-08-31 10:42:22.701853 7fcf45c8ae40 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2017-08-31 10:42:22.701855 7fcf45c8ae40 4 rocksdb: Options.compaction_readahead_size: 0
2017-08-31 10:42:22.701857 7fcf45c8ae40 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-08-31 10:42:22.701860 7fcf45c8ae40 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2017-08-31 10:42:22.701862 7fcf45c8ae40 4 rocksdb: Options.use_adaptive_mutex: 0
2017-08-31 10:42:22.701865 7fcf45c8ae40 4 rocksdb: Options.rate_limiter: (nil)
2017-08-31 10:42:22.701867 7fcf45c8ae40 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-31 10:42:22.701870 7fcf45c8ae40 4 rocksdb: Options.bytes_per_sync: 0
2017-08-31 10:42:22.701872 7fcf45c8ae40 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-08-31 10:42:22.701875 7fcf45c8ae40 4 rocksdb: Options.wal_recovery_mode: 2
2017-08-31 10:42:22.701877 7fcf45c8ae40 4 rocksdb: Options.enable_thread_tracking: 0
2017-08-31 10:42:22.701879 7fcf45c8ae40 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-08-31 10:42:22.701882 7fcf45c8ae40 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-08-31 10:42:22.701885 7fcf45c8ae40 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-08-31 10:42:22.701887 7fcf45c8ae40 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-08-31 10:42:22.701889 7fcf45c8ae40 4 rocksdb: Options.row_cache: None
2017-08-31 10:42:22.701891 7fcf45c8ae40 4 rocksdb: Options.wal_filter: None
2017-08-31 10:42:22.701893 7fcf45c8ae40 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-08-31 10:42:22.701896 7fcf45c8ae40 4 rocksdb: Options.base_background_compactions: 1
2017-08-31 10:42:22.701898 7fcf45c8ae40 4 rocksdb: Options.max_background_compactions: 1
2017-08-31 10:42:22.701900 7fcf45c8ae40 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-08-31 10:42:22.701902 7fcf45c8ae40 4 rocksdb: Options.delayed_write_rate : 16777216
2017-08-31 10:42:22.701905 7fcf45c8ae40 4 rocksdb: Options.max_total_wal_size: 0
2017-08-31 10:42:22.701907 7fcf45c8ae40 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-08-31 10:42:22.701910 7fcf45c8ae40 4 rocksdb: Options.stats_dump_period_sec: 600
2017-08-31 10:42:22.701912 7fcf45c8ae40 4 rocksdb: Compression algorithms supported:
2017-08-31 10:42:22.701918 7fcf45c8ae40 4 rocksdb: Snappy supported: 0
2017-08-31 10:42:22.701921 7fcf45c8ae40 4 rocksdb: Zlib supported: 0
2017-08-31 10:42:22.701924 7fcf45c8ae40 4 rocksdb: Bzip supported: 0
2017-08-31 10:42:22.701926 7fcf45c8ae40 4 rocksdb: LZ4 supported: 0
2017-08-31 10:42:22.701929 7fcf45c8ae40 4 rocksdb: ZSTD supported: 0
2017-08-31 10:42:22.701935 7fcf45c8ae40 4 rocksdb: Fast CRC32 supported: 1
2017-08-31 10:42:22.703074 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-235767

2017-08-31 10:42:22.703470 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

2017-08-31 10:42:22.703498 7fcf45c8ae40 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-08-31 10:42:22.703503 7fcf45c8ae40 4 rocksdb: Options.merge_operator:
2017-08-31 10:42:22.703506 7fcf45c8ae40 4 rocksdb: Options.compaction_filter: None
2017-08-31 10:42:22.703508 7fcf45c8ae40 4 rocksdb: Options.compaction_filter_factory: None
2017-08-31 10:42:22.703511 7fcf45c8ae40 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-08-31 10:42:22.703513 7fcf45c8ae40 4 rocksdb: Options.table_factory: BlockBasedTable
2017-08-31 10:42:22.703552 7fcf45c8ae40 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fcf4fd50158)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7fcf4fd723f8
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2

2017-08-31 10:42:22.703561 7fcf45c8ae40 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-31 10:42:22.703565 7fcf45c8ae40 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-31 10:42:22.703569 7fcf45c8ae40 4 rocksdb: Options.compression: NoCompression
2017-08-31 10:42:22.703572 7fcf45c8ae40 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-31 10:42:22.703574 7fcf45c8ae40 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-31 10:42:22.703578 7fcf45c8ae40 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-31 10:42:22.703580 7fcf45c8ae40 4 rocksdb: Options.num_levels: 7
2017-08-31 10:42:22.703583 7fcf45c8ae40 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-31 10:42:22.703586 7fcf45c8ae40 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-31 10:42:22.703589 7fcf45c8ae40 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-31 10:42:22.703591 7fcf45c8ae40 4 rocksdb: Options.compression_opts.level: -1
2017-08-31 10:42:22.703593 7fcf45c8ae40 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-31 10:42:22.703596 7fcf45c8ae40 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-31 10:42:22.703598 7fcf45c8ae40 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-31 10:42:22.703601 7fcf45c8ae40 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-31 10:42:22.703603 7fcf45c8ae40 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-31 10:42:22.703606 7fcf45c8ae40 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-31 10:42:22.703608 7fcf45c8ae40 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-31 10:42:22.703610 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-31 10:42:22.703613 7fcf45c8ae40 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-31 10:42:22.703616 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-31 10:42:22.703626 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-31 10:42:22.703629 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-31 10:42:22.703633 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-31 10:42:22.703636 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-31 10:42:22.703638 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-31 10:42:22.703641 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-31 10:42:22.703643 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-31 10:42:22.703646 7fcf45c8ae40 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-31 10:42:22.703648 7fcf45c8ae40 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-31 10:42:22.703651 7fcf45c8ae40 4 rocksdb: Options.arena_block_size: 4194304
2017-08-31 10:42:22.703653 7fcf45c8ae40 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-31 10:42:22.703657 7fcf45c8ae40 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-31 10:42:22.703659 7fcf45c8ae40 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-31 10:42:22.703662 7fcf45c8ae40 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-31 10:42:22.703665 7fcf45c8ae40 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-31 10:42:22.703668 7fcf45c8ae40 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-31 10:42:22.703671 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-31 10:42:22.703673 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-31 10:42:22.703675 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-31 10:42:22.703679 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-31 10:42:22.703681 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-31 10:42:22.703683 7fcf45c8ae40 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-31 10:42:22.703710 7fcf45c8ae40 4 rocksdb: Options.table_properties_collectors:
2017-08-31 10:42:22.703714 7fcf45c8ae40 4 rocksdb: Options.inplace_update_support: 0
2017-08-31 10:42:22.703716 7fcf45c8ae40 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-31 10:42:22.703718 7fcf45c8ae40 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-31 10:42:22.703722 7fcf45c8ae40 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-31 10:42:22.703725 7fcf45c8ae40 4 rocksdb: Options.bloom_locality: 0
2017-08-31 10:42:22.703728 7fcf45c8ae40 4 rocksdb: Options.max_successive_merges: 0
2017-08-31 10:42:22.703731 7fcf45c8ae40 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-31 10:42:22.703733 7fcf45c8ae40 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-31 10:42:22.703735 7fcf45c8ae40 4 rocksdb: Options.force_consistency_checks: 0
2017-08-31 10:42:22.703738 7fcf45c8ae40 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-31 10:42:22.708796 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph-mon-1/store.db/MANIFEST-235767 succeeded,manifest_file_number is 235767, next_file_number is 235769, last_sequence is 47598584, log_number is 0,prev_log_number is 0,max_column_family is 0

2017-08-31 10:42:22.708852 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 235766

2017-08-31 10:42:22.709048 7fcf45c8ae40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1504176142709025, "job": 1, "event": "recovery_started", "log_files": [235768]}
2017-08-31 10:42:22.709075 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #235768 mode 2
2017-08-31 10:42:22.709449 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 235770

2017-08-31 10:42:22.721309 7fcf45c8ae40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1504176142721265, "job": 1, "event": "recovery_finished"}
2017-08-31 10:42:22.734659 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fcf50120000
2017-08-31 10:42:22.737569 7fcf45c8ae40 0 starting mon.ceph-mon-1 rank 0 at public addr 10.36.1.13:6789/0 at bind addr 10.36.1.13:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-1 fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
2017-08-31 10:42:22.739073 7fcf45c8ae40 0 starting mon.ceph-mon-1 rank 0 at 10.36.1.13:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-1 fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
2017-08-31 10:42:22.740385 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing) e5 preinit fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
2017-08-31 10:42:22.745487 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-31 10:42:22.745530 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2017-08-31 10:42:22.745894 7fcf45c8ae40 0 mon.ceph-mon-1@-1(probing).mds e214 print_map
e214
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: 3
Filesystem 'rigabackup' (3)
fs_name rigabackup
epoch 201
flags c
created 2017-07-26 08:00:03.508645
modified 2017-07-26 08:00:03.508646
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 3882
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
max_mds 1
in 0
up {0=1732076}
failed
damaged
stopped
data_pools [18]
metadata_pool 19
inline_data disabled
balancer
standby_count_wanted 1
1732076: 10.36.1.13:6800/929051782 'ceph-mon-1' mds.0.193 up:active seq 2038
Standby daemons:
4301670: 10.36.1.18:6800/3569902787 'ceph-mon-2' mds.-1.0 up:standby seq 2
4304099: 10.36.1.19:6800/4255366465 'ceph-mon-3' mds.-1.0 up:standby seq 2

2017-08-31 10:42:22.753474 7fcf45c8ae40 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7fcf45c8ae40 time 2017-08-31 10:42:22.746673
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/mon/OSDMonitor.cc: 3130: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7fcf46280d10]
2: (()+0x4e6bd8) [0x7fcf46183bd8]
3: (OSDMonitor::update_from_paxos(bool*)+0x163a) [0x7fcf461a288a]
4: (PaxosService::refresh(bool*)+0x1ae) [0x7fcf46170a0e]
5: (Monitor::refresh_from_paxos(bool*)+0x193) [0x7fcf46041083]
6: (Monitor::init_paxos()+0x115) [0x7fcf460414b5]
7: (Monitor::preinit()+0x99a) [0x7fcf46041eaa]
8: (main()+0x45a2) [0x7fcf45f70582]
9: (__libc_start_main()+0xf5) [0x7fcf42286b35]
10: (()+0x3775ce) [0x7fcf460145ce]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-195> 2017-08-31 10:42:22.598212 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perfcounters_dump hook 0x7fcf4fd54190
-194> 2017-08-31 10:42:22.598256 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command 1 hook 0x7fcf4fd54190
-193> 2017-08-31 10:42:22.598294 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perf dump hook 0x7fcf4fd54190
-192> 2017-08-31 10:42:22.598348 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perfcounters_schema hook 0x7fcf4fd54190
-191> 2017-08-31 10:42:22.598355 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perf histogram dump hook 0x7fcf4fd54190
-190> 2017-08-31 10:42:22.598360 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command 2 hook 0x7fcf4fd54190
-189> 2017-08-31 10:42:22.598365 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perf schema hook 0x7fcf4fd54190
-188> 2017-08-31 10:42:22.598370 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perf histogram schema hook 0x7fcf4fd54190
-187> 2017-08-31 10:42:22.598377 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command perf reset hook 0x7fcf4fd54190
-186> 2017-08-31 10:42:22.598388 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config show hook 0x7fcf4fd54190
-185> 2017-08-31 10:42:22.598395 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config help hook 0x7fcf4fd54190
-184> 2017-08-31 10:42:22.598400 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config set hook 0x7fcf4fd54190
-183> 2017-08-31 10:42:22.598407 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config get hook 0x7fcf4fd54190
-182> 2017-08-31 10:42:22.598417 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config diff hook 0x7fcf4fd54190
-181> 2017-08-31 10:42:22.598422 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command config diff get hook 0x7fcf4fd54190
-180> 2017-08-31 10:42:22.598429 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command log flush hook 0x7fcf4fd54190
-179> 2017-08-31 10:42:22.598435 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command log dump hook 0x7fcf4fd54190
-178> 2017-08-31 10:42:22.598449 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command log reopen hook 0x7fcf4fd54190
-177> 2017-08-31 10:42:22.598465 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command dump_mempools hook 0x7fcf4ff3fa28
-176> 2017-08-31 10:42:22.620627 7fcf45c8ae40 0 set uid:gid to 167:167 (ceph:ceph)
-175> 2017-08-31 10:42:22.620684 7fcf45c8ae40 0 ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), pid 2412
-174> 2017-08-31 10:42:22.621200 7fcf45c8ae40 0 pidfile_write: ignore empty --pid-file
-173> 2017-08-31 10:42:22.681537 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) init /var/run/ceph/ceph-mon.ceph-mon-1.asok
-172> 2017-08-31 10:42:22.681586 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) bind_and_listen /var/run/ceph/ceph-mon.ceph-mon-1.asok
-171> 2017-08-31 10:42:22.682275 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command 0 hook 0x7fcf4fd50170
-170> 2017-08-31 10:42:22.682303 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command version hook 0x7fcf4fd50170
-169> 2017-08-31 10:42:22.682310 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command git_version hook 0x7fcf4fd50170
-168> 2017-08-31 10:42:22.682318 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command help hook 0x7fcf4fd54290
-167> 2017-08-31 10:42:22.682324 7fcf45c8ae40 5 asok(0x7fcf4fdc41c0) register_command get_command_descriptions hook 0x7fcf4fd542a0
-166> 2017-08-31 10:42:22.682441 7fcf40013700 5 asok(0x7fcf4fdc41c0) entry start
-165> 2017-08-31 10:42:22.700474 7fcf45c8ae40 0 load: jerasure load: lrc load: isa
-164> 2017-08-31 10:42:22.700948 7fcf45c8ae40 0 set rocksdb option compression = kNoCompression
-163> 2017-08-31 10:42:22.701036 7fcf45c8ae40 0 set rocksdb option write_buffer_size = 33554432
-162> 2017-08-31 10:42:22.701108 7fcf45c8ae40 0 set rocksdb option compression = kNoCompression
-161> 2017-08-31 10:42:22.701122 7fcf45c8ae40 0 set rocksdb option write_buffer_size = 33554432
-160> 2017-08-31 10:42:22.701462 7fcf45c8ae40 4 rocksdb: RocksDB version: 5.4.0

-159> 2017-08-31 10:42:22.701491 7fcf45c8ae40 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
-158> 2017-08-31 10:42:22.701495 7fcf45c8ae40 4 rocksdb: Compile date Aug 28 2017
-157> 2017-08-31 10:42:22.701501 7fcf45c8ae40 4 rocksdb: DB SUMMARY

-156> 2017-08-31 10:42:22.701682 7fcf45c8ae40 4 rocksdb: CURRENT file: CURRENT

-155> 2017-08-31 10:42:22.701702 7fcf45c8ae40 4 rocksdb: IDENTITY file: IDENTITY

-154> 2017-08-31 10:42:22.701713 7fcf45c8ae40 4 rocksdb: MANIFEST file: MANIFEST-235767 size: 293 Bytes

-153> 2017-08-31 10:42:22.701719 7fcf45c8ae40 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph-mon-1/store.db dir, Total Num: 4, files: 235681.sst 235682.sst 235683.sst 235685.sst

-152> 2017-08-31 10:42:22.701724 7fcf45c8ae40 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph-mon-1/store.db: 235768.log size: 0 ;

-151> 2017-08-31 10:42:22.701730 7fcf45c8ae40 4 rocksdb: Options.error_if_exists: 0
-150> 2017-08-31 10:42:22.701733 7fcf45c8ae40 4 rocksdb: Options.create_if_missing: 0
-149> 2017-08-31 10:42:22.701735 7fcf45c8ae40 4 rocksdb: Options.paranoid_checks: 1
-148> 2017-08-31 10:42:22.701737 7fcf45c8ae40 4 rocksdb: Options.env: 0x7fcf4ed8fec0
-147> 2017-08-31 10:42:22.701740 7fcf45c8ae40 4 rocksdb: Options.info_log: 0x7fcf5001b520
-146> 2017-08-31 10:42:22.701743 7fcf45c8ae40 4 rocksdb: Options.max_open_files: -1
-145> 2017-08-31 10:42:22.701745 7fcf45c8ae40 4 rocksdb: Options.max_file_opening_threads: 16
-144> 2017-08-31 10:42:22.701747 7fcf45c8ae40 4 rocksdb: Options.use_fsync: 0
-143> 2017-08-31 10:42:22.701750 7fcf45c8ae40 4 rocksdb: Options.max_log_file_size: 0
-142> 2017-08-31 10:42:22.701752 7fcf45c8ae40 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-141> 2017-08-31 10:42:22.701755 7fcf45c8ae40 4 rocksdb: Options.log_file_time_to_roll: 0
-140> 2017-08-31 10:42:22.701757 7fcf45c8ae40 4 rocksdb: Options.keep_log_file_num: 1000
-139> 2017-08-31 10:42:22.701760 7fcf45c8ae40 4 rocksdb: Options.recycle_log_file_num: 0
-138> 2017-08-31 10:42:22.701763 7fcf45c8ae40 4 rocksdb: Options.allow_fallocate: 1
-137> 2017-08-31 10:42:22.701766 7fcf45c8ae40 4 rocksdb: Options.allow_mmap_reads: 0
-136> 2017-08-31 10:42:22.701768 7fcf45c8ae40 4 rocksdb: Options.allow_mmap_writes: 0
-135> 2017-08-31 10:42:22.701770 7fcf45c8ae40 4 rocksdb: Options.use_direct_reads: 0
-134> 2017-08-31 10:42:22.701772 7fcf45c8ae40 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
-133> 2017-08-31 10:42:22.701775 7fcf45c8ae40 4 rocksdb: Options.create_missing_column_families: 0
-132> 2017-08-31 10:42:22.701777 7fcf45c8ae40 4 rocksdb: Options.db_log_dir:
-131> 2017-08-31 10:42:22.701780 7fcf45c8ae40 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-ceph-mon-1/store.db
-130> 2017-08-31 10:42:22.701828 7fcf45c8ae40 4 rocksdb: Options.table_cache_numshardbits: 6
-129> 2017-08-31 10:42:22.701832 7fcf45c8ae40 4 rocksdb: Options.max_subcompactions: 1
-128> 2017-08-31 10:42:22.701835 7fcf45c8ae40 4 rocksdb: Options.max_background_flushes: 1
-127> 2017-08-31 10:42:22.701837 7fcf45c8ae40 4 rocksdb: Options.WAL_ttl_seconds: 0
-126> 2017-08-31 10:42:22.701840 7fcf45c8ae40 4 rocksdb: Options.WAL_size_limit_MB: 0
-125> 2017-08-31 10:42:22.701842 7fcf45c8ae40 4 rocksdb: Options.manifest_preallocation_size: 4194304
-124> 2017-08-31 10:42:22.701844 7fcf45c8ae40 4 rocksdb: Options.is_fd_close_on_exec: 1
-123> 2017-08-31 10:42:22.701846 7fcf45c8ae40 4 rocksdb: Options.advise_random_on_open: 1
-122> 2017-08-31 10:42:22.701848 7fcf45c8ae40 4 rocksdb: Options.db_write_buffer_size: 0
-121> 2017-08-31 10:42:22.701851 7fcf45c8ae40 4 rocksdb: Options.access_hint_on_compaction_start: 1
-120> 2017-08-31 10:42:22.701853 7fcf45c8ae40 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-119> 2017-08-31 10:42:22.701855 7fcf45c8ae40 4 rocksdb: Options.compaction_readahead_size: 0
-118> 2017-08-31 10:42:22.701857 7fcf45c8ae40 4 rocksdb: Options.random_access_max_buffer_size: 1048576
-117> 2017-08-31 10:42:22.701860 7fcf45c8ae40 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
-116> 2017-08-31 10:42:22.701862 7fcf45c8ae40 4 rocksdb: Options.use_adaptive_mutex: 0
-115> 2017-08-31 10:42:22.701865 7fcf45c8ae40 4 rocksdb: Options.rate_limiter: (nil)
-114> 2017-08-31 10:42:22.701867 7fcf45c8ae40 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
-113> 2017-08-31 10:42:22.701870 7fcf45c8ae40 4 rocksdb: Options.bytes_per_sync: 0
-112> 2017-08-31 10:42:22.701872 7fcf45c8ae40 4 rocksdb: Options.wal_bytes_per_sync: 0
-111> 2017-08-31 10:42:22.701875 7fcf45c8ae40 4 rocksdb: Options.wal_recovery_mode: 2
-110> 2017-08-31 10:42:22.701877 7fcf45c8ae40 4 rocksdb: Options.enable_thread_tracking: 0
-109> 2017-08-31 10:42:22.701879 7fcf45c8ae40 4 rocksdb: Options.allow_concurrent_memtable_write: 1
-108> 2017-08-31 10:42:22.701882 7fcf45c8ae40 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
-107> 2017-08-31 10:42:22.701885 7fcf45c8ae40 4 rocksdb: Options.write_thread_max_yield_usec: 100
-106> 2017-08-31 10:42:22.701887 7fcf45c8ae40 4 rocksdb: Options.write_thread_slow_yield_usec: 3
-105> 2017-08-31 10:42:22.701889 7fcf45c8ae40 4 rocksdb: Options.row_cache: None
-104> 2017-08-31 10:42:22.701891 7fcf45c8ae40 4 rocksdb: Options.wal_filter: None
-103> 2017-08-31 10:42:22.701893 7fcf45c8ae40 4 rocksdb: Options.avoid_flush_during_recovery: 0
-102> 2017-08-31 10:42:22.701896 7fcf45c8ae40 4 rocksdb: Options.base_background_compactions: 1
-101> 2017-08-31 10:42:22.701898 7fcf45c8ae40 4 rocksdb: Options.max_background_compactions: 1
-100> 2017-08-31 10:42:22.701900 7fcf45c8ae40 4 rocksdb: Options.avoid_flush_during_shutdown: 0
-99> 2017-08-31 10:42:22.701902 7fcf45c8ae40 4 rocksdb: Options.delayed_write_rate : 16777216
-98> 2017-08-31 10:42:22.701905 7fcf45c8ae40 4 rocksdb: Options.max_total_wal_size: 0
-97> 2017-08-31 10:42:22.701907 7fcf45c8ae40 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-96> 2017-08-31 10:42:22.701910 7fcf45c8ae40 4 rocksdb: Options.stats_dump_period_sec: 600
-95> 2017-08-31 10:42:22.701912 7fcf45c8ae40 4 rocksdb: Compression algorithms supported:
-94> 2017-08-31 10:42:22.701918 7fcf45c8ae40 4 rocksdb: Snappy supported: 0
-93> 2017-08-31 10:42:22.701921 7fcf45c8ae40 4 rocksdb: Zlib supported: 0
-92> 2017-08-31 10:42:22.701924 7fcf45c8ae40 4 rocksdb: Bzip supported: 0
-91> 2017-08-31 10:42:22.701926 7fcf45c8ae40 4 rocksdb: LZ4 supported: 0
-90> 2017-08-31 10:42:22.701929 7fcf45c8ae40 4 rocksdb: ZSTD supported: 0
-89> 2017-08-31 10:42:22.701935 7fcf45c8ae40 4 rocksdb: Fast CRC32 supported: 1
-88> 2017-08-31 10:42:22.703074 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-235767

-87> 2017-08-31 10:42:22.703470 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

-86> 2017-08-31 10:42:22.703498 7fcf45c8ae40 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
-85> 2017-08-31 10:42:22.703503 7fcf45c8ae40 4 rocksdb: Options.merge_operator:
-84> 2017-08-31 10:42:22.703506 7fcf45c8ae40 4 rocksdb: Options.compaction_filter: None
-83> 2017-08-31 10:42:22.703508 7fcf45c8ae40 4 rocksdb: Options.compaction_filter_factory: None
-82> 2017-08-31 10:42:22.703511 7fcf45c8ae40 4 rocksdb: Options.memtable_factory: SkipListFactory
-81> 2017-08-31 10:42:22.703513 7fcf45c8ae40 4 rocksdb: Options.table_factory: BlockBasedTable
-80> 2017-08-31 10:42:22.703552 7fcf45c8ae40 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fcf4fd50158)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7fcf4fd723f8
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2

-79> 2017-08-31 10:42:22.703561 7fcf45c8ae40 4 rocksdb: Options.write_buffer_size: 33554432
-78> 2017-08-31 10:42:22.703565 7fcf45c8ae40 4 rocksdb: Options.max_write_buffer_number: 2
-77> 2017-08-31 10:42:22.703569 7fcf45c8ae40 4 rocksdb: Options.compression: NoCompression
-76> 2017-08-31 10:42:22.703572 7fcf45c8ae40 4 rocksdb: Options.bottommost_compression: Disabled
-75> 2017-08-31 10:42:22.703574 7fcf45c8ae40 4 rocksdb: Options.prefix_extractor: nullptr
-74> 2017-08-31 10:42:22.703578 7fcf45c8ae40 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
-73> 2017-08-31 10:42:22.703580 7fcf45c8ae40 4 rocksdb: Options.num_levels: 7
-72> 2017-08-31 10:42:22.703583 7fcf45c8ae40 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
-71> 2017-08-31 10:42:22.703586 7fcf45c8ae40 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-70> 2017-08-31 10:42:22.703589 7fcf45c8ae40 4 rocksdb: Options.compression_opts.window_bits: -14
-69> 2017-08-31 10:42:22.703591 7fcf45c8ae40 4 rocksdb: Options.compression_opts.level: -1
-68> 2017-08-31 10:42:22.703593 7fcf45c8ae40 4 rocksdb: Options.compression_opts.strategy: 0
-67> 2017-08-31 10:42:22.703596 7fcf45c8ae40 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
-66> 2017-08-31 10:42:22.703598 7fcf45c8ae40 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-65> 2017-08-31 10:42:22.703601 7fcf45c8ae40 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-64> 2017-08-31 10:42:22.703603 7fcf45c8ae40 4 rocksdb: Options.level0_stop_writes_trigger: 36
-63> 2017-08-31 10:42:22.703606 7fcf45c8ae40 4 rocksdb: Options.target_file_size_base: 67108864
-62> 2017-08-31 10:42:22.703608 7fcf45c8ae40 4 rocksdb: Options.target_file_size_multiplier: 1
-61> 2017-08-31 10:42:22.703610 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_base: 268435456
-60> 2017-08-31 10:42:22.703613 7fcf45c8ae40 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-59> 2017-08-31 10:42:22.703616 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
-58> 2017-08-31 10:42:22.703626 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-57> 2017-08-31 10:42:22.703629 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-56> 2017-08-31 10:42:22.703633 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-55> 2017-08-31 10:42:22.703636 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-54> 2017-08-31 10:42:22.703638 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-53> 2017-08-31 10:42:22.703641 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-52> 2017-08-31 10:42:22.703643 7fcf45c8ae40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-51> 2017-08-31 10:42:22.703646 7fcf45c8ae40 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-50> 2017-08-31 10:42:22.703648 7fcf45c8ae40 4 rocksdb: Options.max_compaction_bytes: 1677721600
-49> 2017-08-31 10:42:22.703651 7fcf45c8ae40 4 rocksdb: Options.arena_block_size: 4194304
-48> 2017-08-31 10:42:22.703653 7fcf45c8ae40 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
-47> 2017-08-31 10:42:22.703657 7fcf45c8ae40 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
-46> 2017-08-31 10:42:22.703659 7fcf45c8ae40 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
-45> 2017-08-31 10:42:22.703662 7fcf45c8ae40 4 rocksdb: Options.disable_auto_compactions: 0
-44> 2017-08-31 10:42:22.703665 7fcf45c8ae40 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
-43> 2017-08-31 10:42:22.703668 7fcf45c8ae40 4 rocksdb: Options.compaction_pri: kByCompensatedSize
-42> 2017-08-31 10:42:22.703671 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-41> 2017-08-31 10:42:22.703673 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-40> 2017-08-31 10:42:22.703675 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-39> 2017-08-31 10:42:22.703679 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-38> 2017-08-31 10:42:22.703681 7fcf45c8ae40 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-37> 2017-08-31 10:42:22.703683 7fcf45c8ae40 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-36> 2017-08-31 10:42:22.703710 7fcf45c8ae40 4 rocksdb: Options.table_properties_collectors:
-35> 2017-08-31 10:42:22.703714 7fcf45c8ae40 4 rocksdb: Options.inplace_update_support: 0
-34> 2017-08-31 10:42:22.703716 7fcf45c8ae40 4 rocksdb: Options.inplace_update_num_locks: 10000
-33> 2017-08-31 10:42:22.703718 7fcf45c8ae40 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-32> 2017-08-31 10:42:22.703722 7fcf45c8ae40 4 rocksdb: Options.memtable_huge_page_size: 0
-31> 2017-08-31 10:42:22.703725 7fcf45c8ae40 4 rocksdb: Options.bloom_locality: 0
-30> 2017-08-31 10:42:22.703728 7fcf45c8ae40 4 rocksdb: Options.max_successive_merges: 0
-29> 2017-08-31 10:42:22.703731 7fcf45c8ae40 4 rocksdb: Options.optimize_filters_for_hits: 0
-28> 2017-08-31 10:42:22.703733 7fcf45c8ae40 4 rocksdb: Options.paranoid_file_checks: 0
-27> 2017-08-31 10:42:22.703735 7fcf45c8ae40 4 rocksdb: Options.force_consistency_checks: 0
-26> 2017-08-31 10:42:22.703738 7fcf45c8ae40 4 rocksdb: Options.report_bg_io_stats: 0
-25> 2017-08-31 10:42:22.708796 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph-mon-1/store.db/MANIFEST-235767 succeeded,manifest_file_number is 235767, next_file_number is 235769, last_sequence is 47598584, log_number is 0,prev_log_number is 0,max_column_family is 0

-24> 2017-08-31 10:42:22.708852 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 235766

-23> 2017-08-31 10:42:22.709048 7fcf45c8ae40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1504176142709025, "job": 1, "event": "recovery_started", "log_files": [235768]}
-22> 2017-08-31 10:42:22.709075 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #235768 mode 2
-21> 2017-08-31 10:42:22.709449 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 235770

-20> 2017-08-31 10:42:22.721309 7fcf45c8ae40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1504176142721265, "job": 1, "event": "recovery_finished"}
-19> 2017-08-31 10:42:22.722075 7fcf45c8ae40 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph-mon-1/store.db//MANIFEST-235767 type=3 #235767 -- OK

-18> 2017-08-31 10:42:22.722161 7fcf45c8ae40 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph-mon-1/store.db//235768.log type=0 #235768 -- OK

-17> 2017-08-31 10:42:22.734659 7fcf45c8ae40 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fcf50120000
-16> 2017-08-31 10:42:22.737316 7fcf37fae700 2 Event(0x7fcf4fdc2a80 nevent=5000 time_id=1).set_owner idx=0 owner=140527974147840
-15> 2017-08-31 10:42:22.737352 7fcf387af700 2 Event(0x7fcf4fdc3280 nevent=5000 time_id=1).set_owner idx=1 owner=140527982540544
-14> 2017-08-31 10:42:22.737389 7fcf38fb0700 2 Event(0x7fcf4fdc1a80 nevent=5000 time_id=1).set_owner idx=2 owner=140527990933248
-13> 2017-08-31 10:42:22.737569 7fcf45c8ae40 0 starting mon.ceph-mon-1 rank 0 at public addr 10.36.1.13:6789/0 at bind addr 10.36.1.13:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-1 fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
-12> 2017-08-31 10:42:22.738959 7fcf45c8ae40 1 -- 10.36.1.13:6789/0 learned_addr learned my addr 10.36.1.13:6789/0
-11> 2017-08-31 10:42:22.739017 7fcf45c8ae40 1 -- 10.36.1.13:6789/0 _finish_bind bind my_inst.addr is 10.36.1.13:6789/0
-10> 2017-08-31 10:42:22.739073 7fcf45c8ae40 0 starting mon.ceph-mon-1 rank 0 at 10.36.1.13:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-1 fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
-9> 2017-08-31 10:42:22.739138 7fcf45c8ae40 5 adding auth protocol: cephx
-8> 2017-08-31 10:42:22.739146 7fcf45c8ae40 5 adding auth protocol: cephx
-7> 2017-08-31 10:42:22.739211 7fcf45c8ae40 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-6> 2017-08-31 10:42:22.739236 7fcf45c8ae40 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-5> 2017-08-31 10:42:22.740385 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing) e5 preinit fsid bc05949d-98ba-4006-a0f5-ca2eabd5b18d
-4> 2017-08-31 10:42:22.745487 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
-3> 2017-08-31 10:42:22.745530 7fcf45c8ae40 1 mon.ceph-mon-1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
-2> 2017-08-31 10:42:22.745869 7fcf45c8ae40 4 mon.ceph-mon-1@-1(probing).mds e214 new map
-1> 2017-08-31 10:42:22.745894 7fcf45c8ae40 0 mon.ceph-mon-1@-1(probing).mds e214 print_map
e214
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: 3
Filesystem 'rigabackup' (3)
fs_name rigabackup
epoch 201
flags c
created 2017-07-26 08:00:03.508645
modified 2017-07-26 08:00:03.508646
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 3882
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
max_mds 1
in 0
up {0=1732076}
failed
damaged
stopped
data_pools [18]
metadata_pool 19
inline_data disabled
balancer
standby_count_wanted 1
1732076: 10.36.1.13:6800/929051782 'ceph-mon-1' mds.0.193 up:active seq 2038
Standby daemons:
4301670: 10.36.1.18:6800/3569902787 'ceph-mon-2' mds.-1.0 up:standby seq 2
4304099: 10.36.1.19:6800/4255366465 'ceph-mon-3' mds.-1.0 up:standby seq 2

0> 2017-08-31 10:42:22.753474 7fcf45c8ae40 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7fcf45c8ae40 time 2017-08-31 10:42:22.746673
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/mon/OSDMonitor.cc: 3130: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7fcf46280d10]
2: (()+0x4e6bd8) [0x7fcf46183bd8]
3: (OSDMonitor::update_from_paxos(bool*)+0x163a) [0x7fcf461a288a]
4: (PaxosService::refresh(bool*)+0x1ae) [0x7fcf46170a0e]
5: (Monitor::refresh_from_paxos(bool*)+0x193) [0x7fcf46041083]
6: (Monitor::init_paxos()+0x115) [0x7fcf460414b5]
7: (Monitor::preinit()+0x99a) [0x7fcf46041eaa]
8: (main()+0x45a2) [0x7fcf45f70582]
9: (__libc_start_main()+0xf5) [0x7fcf42286b35]
10: (()+0x3775ce) [0x7fcf460145ce]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph-mon-1.log
--- end dump of recent events ---
2017-08-31 10:42:22.761086 7fcf45c8ae40 -1 *** Caught signal (Aborted) **
in thread 7fcf45c8ae40 thread_name:ceph-mon

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0x8e5cc1) [0x7fcf46582cc1]
2: (()+0xf370) [0x7fcf4501c370]
3: (gsignal()+0x37) [0x7fcf4229a1d7]
4: (abort()+0x148) [0x7fcf4229b8c8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7fcf46280e84]
6: (()+0x4e6bd8) [0x7fcf46183bd8]
7: (OSDMonitor::update_from_paxos(bool*)+0x163a) [0x7fcf461a288a]
8: (PaxosService::refresh(bool*)+0x1ae) [0x7fcf46170a0e]
9: (Monitor::refresh_from_paxos(bool*)+0x193) [0x7fcf46041083]
10: (Monitor::init_paxos()+0x115) [0x7fcf460414b5]
11: (Monitor::preinit()+0x99a) [0x7fcf46041eaa]
12: (main()+0x45a2) [0x7fcf45f70582]
13: (__libc_start_main()+0xf5) [0x7fcf42286b35]
14: (()+0x3775ce) [0x7fcf460145ce]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2017-08-31 10:42:22.761086 7fcf45c8ae40 -1 *** Caught signal (Aborted) **
in thread 7fcf45c8ae40 thread_name:ceph-mon

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0x8e5cc1) [0x7fcf46582cc1]
2: (()+0xf370) [0x7fcf4501c370]
3: (gsignal()+0x37) [0x7fcf4229a1d7]
4: (abort()+0x148) [0x7fcf4229b8c8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7fcf46280e84]
6: (()+0x4e6bd8) [0x7fcf46183bd8]
7: (OSDMonitor::update_from_paxos(bool*)+0x163a) [0x7fcf461a288a]
8: (PaxosService::refresh(bool*)+0x1ae) [0x7fcf46170a0e]
9: (Monitor::refresh_from_paxos(bool*)+0x193) [0x7fcf46041083]
10: (Monitor::init_paxos()+0x115) [0x7fcf460414b5]
11: (Monitor::preinit()+0x99a) [0x7fcf46041eaa]
12: (main()+0x45a2) [0x7fcf45f70582]
13: (__libc_start_main()+0xf5) [0x7fcf42286b35]
14: (()+0x3775ce) [0x7fcf460145ce]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph-mon-1.log
--- end dump of recent events ---
(2-2/2)