Actions
Support #22680
closedmons segmentation faults New 12.2.2 cluster
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Component(RADOS):
Monitor
Pull request ID:
Description
Hi all,
I installed a new Luminous 12.2.2 cluster. The monitors were up at first, but quickly started failing, segfaulting.
I only installed some mons, mgr, mds with ceph-deploy and osds with ceph volume. No pools or fs were created yet.
When I start all mons again, there is a short window i can see the cluster state:
[root@ceph001 ~]# ceph status
cluster:
id: 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
health: HEALTH_WARN
1/3 mons down, quorum ceph002,ceph003
services:
mon: 3 daemons, quorum ceph002,ceph003, out of quorum: ceph001
mgr: ceph001(active), standbys: ceph002, ceph003
osd: 7 osds: 4 up, 4 indata:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 4223 MB used, 14899 GB / 14904 GB avail
pgs:But this is only until I lose quorum again.
What could be the problem here?
Thanks!!
Kenneth
2018-01-12 13:08:36.912832 7f794f513e80 0 set uid:gid to 167:167 (ceph:ceph)
2018-01-12 13:08:36.912859 7f794f513e80 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 28726
2018-01-12 13:08:36.913016 7f794f513e80 0 pidfile_write: ignore empty --pid-file
2018-01-12 13:08:36.951556 7f794f513e80 0 load: jerasure load: lrc load: isa
2018-01-12 13:08:36.951703 7f794f513e80 0 set rocksdb option compression = kNoCompression
2018-01-12 13:08:36.951716 7f794f513e80 0 set rocksdb option write_buffer_size = 33554432
2018-01-12 13:08:36.951742 7f794f513e80 0 set rocksdb option compression = kNoCompression
2018-01-12 13:08:36.951749 7f794f513e80 0 set rocksdb option write_buffer_size = 33554432
2018-01-12 13:08:36.951936 7f794f513e80 4 rocksdb: RocksDB version: 5.4.0
2018-01-12 13:08:36.951947 7f794f513e80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-01-12 13:08:36.951951 7f794f513e80 4 rocksdb: Compile date Nov 30 2017
2018-01-12 13:08:36.951954 7f794f513e80 4 rocksdb: DB SUMMARY
2018-01-12 13:08:36.952011 7f794f513e80 4 rocksdb: CURRENT file: CURRENT
2018-01-12 13:08:36.952016 7f794f513e80 4 rocksdb: IDENTITY file: IDENTITY
2018-01-12 13:08:36.952020 7f794f513e80 4 rocksdb: MANIFEST file: MANIFEST-000064 size: 219 Bytes
2018-01-12 13:08:36.952023 7f794f513e80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph001/store.db dir, Total Num: 3, files: 000048.sst 000050.sst 000060.sst
2018-01-12 13:08:36.952025 7f794f513e80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph001/store.db: 000065.log size: 0 ;
2018-01-12 13:08:36.952028 7f794f513e80 4 rocksdb: Options.error_if_exists: 0
2018-01-12 13:08:36.952029 7f794f513e80 4 rocksdb: Options.create_if_missing: 0
2018-01-12 13:08:36.952031 7f794f513e80 4 rocksdb: Options.paranoid_checks: 1
2018-01-12 13:08:36.952032 7f794f513e80 4 rocksdb: Options.env: 0x5617a10fa040
2018-01-12 13:08:36.952033 7f794f513e80 4 rocksdb: Options.info_log: 0x5617a24ce1c0
2018-01-12 13:08:36.952034 7f794f513e80 4 rocksdb: Options.max_open_files: -1
2018-01-12 13:08:36.952035 7f794f513e80 4 rocksdb: Options.max_file_opening_threads: 16
2018-01-12 13:08:36.952035 7f794f513e80 4 rocksdb: Options.use_fsync: 0
2018-01-12 13:08:36.952037 7f794f513e80 4 rocksdb: Options.max_log_file_size: 0
2018-01-12 13:08:36.952038 7f794f513e80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2018-01-12 13:08:36.952039 7f794f513e80 4 rocksdb: Options.log_file_time_to_roll: 0
2018-01-12 13:08:36.952040 7f794f513e80 4 rocksdb: Options.keep_log_file_num: 1000
2018-01-12 13:08:36.952041 7f794f513e80 4 rocksdb: Options.recycle_log_file_num: 0
2018-01-12 13:08:36.952042 7f794f513e80 4 rocksdb: Options.allow_fallocate: 1
2018-01-12 13:08:36.952043 7f794f513e80 4 rocksdb: Options.allow_mmap_reads: 0
2018-01-12 13:08:36.952044 7f794f513e80 4 rocksdb: Options.allow_mmap_writes: 0
2018-01-12 13:08:36.952045 7f794f513e80 4 rocksdb: Options.use_direct_reads: 0
2018-01-12 13:08:36.952046 7f794f513e80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2018-01-12 13:08:36.952047 7f794f513e80 4 rocksdb: Options.create_missing_column_families: 0
2018-01-12 13:08:36.952048 7f794f513e80 4 rocksdb: Options.db_log_dir:
2018-01-12 13:08:36.952049 7f794f513e80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-ceph001/store.db
2018-01-12 13:08:36.952050 7f794f513e80 4 rocksdb: Options.table_cache_numshardbits: 6
2018-01-12 13:08:36.952050 7f794f513e80 4 rocksdb: Options.max_subcompactions: 1
2018-01-12 13:08:36.952062 7f794f513e80 4 rocksdb: Options.max_background_flushes: 1
2018-01-12 13:08:36.952063 7f794f513e80 4 rocksdb: Options.WAL_ttl_seconds: 0
2018-01-12 13:08:36.952064 7f794f513e80 4 rocksdb: Options.WAL_size_limit_MB: 0
2018-01-12 13:08:36.952065 7f794f513e80 4 rocksdb: Options.manifest_preallocation_size: 4194304
2018-01-12 13:08:36.952066 7f794f513e80 4 rocksdb: Options.is_fd_close_on_exec: 1
2018-01-12 13:08:36.952067 7f794f513e80 4 rocksdb: Options.advise_random_on_open: 1
2018-01-12 13:08:36.952068 7f794f513e80 4 rocksdb: Options.db_write_buffer_size: 0
2018-01-12 13:08:36.952069 7f794f513e80 4 rocksdb: Options.access_hint_on_compaction_start: 1
2018-01-12 13:08:36.952069 7f794f513e80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2018-01-12 13:08:36.952070 7f794f513e80 4 rocksdb: Options.compaction_readahead_size: 0
2018-01-12 13:08:36.952071 7f794f513e80 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2018-01-12 13:08:36.952072 7f794f513e80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2018-01-12 13:08:36.952073 7f794f513e80 4 rocksdb: Options.use_adaptive_mutex: 0
2018-01-12 13:08:36.952074 7f794f513e80 4 rocksdb: Options.rate_limiter: (nil)
2018-01-12 13:08:36.952075 7f794f513e80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2018-01-12 13:08:36.952076 7f794f513e80 4 rocksdb: Options.bytes_per_sync: 0
2018-01-12 13:08:36.952077 7f794f513e80 4 rocksdb: Options.wal_bytes_per_sync: 0
2018-01-12 13:08:36.952078 7f794f513e80 4 rocksdb: Options.wal_recovery_mode: 2
2018-01-12 13:08:36.952078 7f794f513e80 4 rocksdb: Options.enable_thread_tracking: 0
2018-01-12 13:08:36.952079 7f794f513e80 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2018-01-12 13:08:36.952080 7f794f513e80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2018-01-12 13:08:36.952081 7f794f513e80 4 rocksdb: Options.write_thread_max_yield_usec: 100
2018-01-12 13:08:36.952082 7f794f513e80 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2018-01-12 13:08:36.952083 7f794f513e80 4 rocksdb: Options.row_cache: None
2018-01-12 13:08:36.952083 7f794f513e80 4 rocksdb: Options.wal_filter: None
2018-01-12 13:08:36.952084 7f794f513e80 4 rocksdb: Options.avoid_flush_during_recovery: 0
2018-01-12 13:08:36.952086 7f794f513e80 4 rocksdb: Options.base_background_compactions: 1
2018-01-12 13:08:36.952087 7f794f513e80 4 rocksdb: Options.max_background_compactions: 1
2018-01-12 13:08:36.952088 7f794f513e80 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2018-01-12 13:08:36.952088 7f794f513e80 4 rocksdb: Options.delayed_write_rate : 16777216
2018-01-12 13:08:36.952089 7f794f513e80 4 rocksdb: Options.max_total_wal_size: 0
2018-01-12 13:08:36.952091 7f794f513e80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2018-01-12 13:08:36.952092 7f794f513e80 4 rocksdb: Options.stats_dump_period_sec: 600
2018-01-12 13:08:36.952093 7f794f513e80 4 rocksdb: Compression algorithms supported:
2018-01-12 13:08:36.952096 7f794f513e80 4 rocksdb: Snappy supported: 0
2018-01-12 13:08:36.952097 7f794f513e80 4 rocksdb: Zlib supported: 0
2018-01-12 13:08:36.952098 7f794f513e80 4 rocksdb: Bzip supported: 0
2018-01-12 13:08:36.952099 7f794f513e80 4 rocksdb: LZ4 supported: 0
2018-01-12 13:08:36.952100 7f794f513e80 4 rocksdb: ZSTD supported: 0
2018-01-12 13:08:36.952102 7f794f513e80 4 rocksdb: Fast CRC32 supported: 0
2018-01-12 13:08:36.952354 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000064
2018-01-12 13:08:36.952489 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
2018-01-12 13:08:36.952496 7f794f513e80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2018-01-12 13:08:36.952498 7f794f513e80 4 rocksdb: Options.merge_operator:
2018-01-12 13:08:36.952502 7f794f513e80 4 rocksdb: Options.compaction_filter: None
2018-01-12 13:08:36.952503 7f794f513e80 4 rocksdb: Options.compaction_filter_factory: None
2018-01-12 13:08:36.952504 7f794f513e80 4 rocksdb: Options.memtable_factory: SkipListFactory
2018-01-12 13:08:36.952505 7f794f513e80 4 rocksdb: Options.table_factory: BlockBasedTable
2018-01-12 13:08:36.952525 7f794f513e80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617a23e80a0)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x5617a24068c8
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
2018-01-12 13:08:36.952547 7f794f513e80 4 rocksdb: Options.write_buffer_size: 33554432
2018-01-12 13:08:36.952548 7f794f513e80 4 rocksdb: Options.max_write_buffer_number: 2
2018-01-12 13:08:36.952550 7f794f513e80 4 rocksdb: Options.compression: NoCompression
2018-01-12 13:08:36.952561 7f794f513e80 4 rocksdb: Options.bottommost_compression: Disabled
2018-01-12 13:08:36.952563 7f794f513e80 4 rocksdb: Options.prefix_extractor: nullptr
2018-01-12 13:08:36.952564 7f794f513e80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-01-12 13:08:36.952565 7f794f513e80 4 rocksdb: Options.num_levels: 7
2018-01-12 13:08:36.952565 7f794f513e80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2018-01-12 13:08:36.952567 7f794f513e80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2018-01-12 13:08:36.952568 7f794f513e80 4 rocksdb: Options.compression_opts.window_bits: -14
2018-01-12 13:08:36.952568 7f794f513e80 4 rocksdb: Options.compression_opts.level: -1
2018-01-12 13:08:36.952569 7f794f513e80 4 rocksdb: Options.compression_opts.strategy: 0
2018-01-12 13:08:36.952570 7f794f513e80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2018-01-12 13:08:36.952571 7f794f513e80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2018-01-12 13:08:36.952573 7f794f513e80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2018-01-12 13:08:36.952573 7f794f513e80 4 rocksdb: Options.level0_stop_writes_trigger: 36
2018-01-12 13:08:36.952574 7f794f513e80 4 rocksdb: Options.target_file_size_base: 67108864
2018-01-12 13:08:36.952575 7f794f513e80 4 rocksdb: Options.target_file_size_multiplier: 1
2018-01-12 13:08:36.952576 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2018-01-12 13:08:36.952577 7f794f513e80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2018-01-12 13:08:36.952578 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2018-01-12 13:08:36.952583 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-01-12 13:08:36.952584 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-01-12 13:08:36.952585 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-01-12 13:08:36.952586 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-01-12 13:08:36.952588 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-01-12 13:08:36.952589 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-01-12 13:08:36.952589 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-01-12 13:08:36.952590 7f794f513e80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2018-01-12 13:08:36.952591 7f794f513e80 4 rocksdb: Options.max_compaction_bytes: 1677721600
2018-01-12 13:08:36.952592 7f794f513e80 4 rocksdb: Options.arena_block_size: 4194304
2018-01-12 13:08:36.952593 7f794f513e80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2018-01-12 13:08:36.952595 7f794f513e80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2018-01-12 13:08:36.952595 7f794f513e80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2018-01-12 13:08:36.952597 7f794f513e80 4 rocksdb: Options.disable_auto_compactions: 0
2018-01-12 13:08:36.952598 7f794f513e80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2018-01-12 13:08:36.952599 7f794f513e80 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2018-01-12 13:08:36.952600 7f794f513e80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2018-01-12 13:08:36.952601 7f794f513e80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2018-01-12 13:08:36.952602 7f794f513e80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2018-01-12 13:08:36.952603 7f794f513e80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2018-01-12 13:08:36.952604 7f794f513e80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2018-01-12 13:08:36.952605 7f794f513e80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-01-12 13:08:36.952616 7f794f513e80 4 rocksdb: Options.table_properties_collectors:
2018-01-12 13:08:36.952618 7f794f513e80 4 rocksdb: Options.inplace_update_support: 0
2018-01-12 13:08:36.952619 7f794f513e80 4 rocksdb: Options.inplace_update_num_locks: 10000
2018-01-12 13:08:36.952620 7f794f513e80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-01-12 13:08:36.952621 7f794f513e80 4 rocksdb: Options.memtable_huge_page_size: 0
2018-01-12 13:08:36.952622 7f794f513e80 4 rocksdb: Options.bloom_locality: 0
2018-01-12 13:08:36.952623 7f794f513e80 4 rocksdb: Options.max_successive_merges: 0
2018-01-12 13:08:36.952624 7f794f513e80 4 rocksdb: Options.optimize_filters_for_hits: 0
2018-01-12 13:08:36.952625 7f794f513e80 4 rocksdb: Options.paranoid_file_checks: 0
2018-01-12 13:08:36.952626 7f794f513e80 4 rocksdb: Options.force_consistency_checks: 0
2018-01-12 13:08:36.952627 7f794f513e80 4 rocksdb: Options.report_bg_io_stats: 0
2018-01-12 13:08:36.955246 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph001/store.db/MANIFEST-000064 succeeded,manifest_file_number is 64, next_file_number is 66, last_sequence is 10347, log_number is 0,prev_log_number is 0,max_column_family is 0
2018-01-12 13:08:36.955257 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 63
2018-01-12 13:08:36.955357 7f794f513e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916955344, "job": 1, "event": "recovery_started", "log_files": [65]}
2018-01-12 13:08:36.955365 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #65 mode 2
2018-01-12 13:08:36.955496 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 67
2018-01-12 13:08:36.956689 7f794f513e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916956669, "job": 1, "event": "recovery_finished"}
2018-01-12 13:08:36.959779 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5617a26a4000
2018-01-12 13:08:36.964040 7f794f513e80 -1 *** Caught signal (Segmentation fault) **
in thread 7f794f513e80 thread_name:ceph-mon
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0x8eae11) [0x5617988d4e11]
2: (()+0xf5e0) [0x7f794e6765e0]
3: (malloc_usable_size()+0x8a) [0x7f794f0d8b7a]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x5617989a1eff]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x427) [0x5617989a3057]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5617989a32d6]
7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x5617989a3876]
8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x5617989a619f]
9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x2a4) [0x561798a635c4]
10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x561798969050]
11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x561798a10f04]
12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x561798a11489]
13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x561798a154a5]
14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x561798a1494a]
15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x561798491a42]
16: (main()+0xca7) [0x5617982bae37]
17: (__libc_start_main()+0xf5) [0x7f794baf8c05]
18: (()+0x378e1e) [0x561798362e1e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-179> 2018-01-12 13:08:36.904404 7f794f513e80 5 asok(0x5617a245c1c0) register_command perfcounters_dump hook 0x5617a23ec180
-178> 2018-01-12 13:08:36.904434 7f794f513e80 5 asok(0x5617a245c1c0) register_command 1 hook 0x5617a23ec180
-177> 2018-01-12 13:08:36.904440 7f794f513e80 5 asok(0x5617a245c1c0) register_command perf dump hook 0x5617a23ec180
-176> 2018-01-12 13:08:36.904442 7f794f513e80 5 asok(0x5617a245c1c0) register_command perfcounters_schema hook 0x5617a23ec180
-175> 2018-01-12 13:08:36.904448 7f794f513e80 5 asok(0x5617a245c1c0) register_command perf histogram dump hook 0x5617a23ec180
-174> 2018-01-12 13:08:36.904450 7f794f513e80 5 asok(0x5617a245c1c0) register_command 2 hook 0x5617a23ec180
-173> 2018-01-12 13:08:36.904468 7f794f513e80 5 asok(0x5617a245c1c0) register_command perf schema hook 0x5617a23ec180
-172> 2018-01-12 13:08:36.904473 7f794f513e80 5 asok(0x5617a245c1c0) register_command perf histogram schema hook 0x5617a23ec180
-171> 2018-01-12 13:08:36.904485 7f794f513e80 5 asok(0x5617a245c1c0) register_command perf reset hook 0x5617a23ec180
-170> 2018-01-12 13:08:36.904492 7f794f513e80 5 asok(0x5617a245c1c0) register_command config show hook 0x5617a23ec180
-169> 2018-01-12 13:08:36.904496 7f794f513e80 5 asok(0x5617a245c1c0) register_command config help hook 0x5617a23ec180
-168> 2018-01-12 13:08:36.904502 7f794f513e80 5 asok(0x5617a245c1c0) register_command config set hook 0x5617a23ec180
-167> 2018-01-12 13:08:36.904505 7f794f513e80 5 asok(0x5617a245c1c0) register_command config get hook 0x5617a23ec180
-166> 2018-01-12 13:08:36.904509 7f794f513e80 5 asok(0x5617a245c1c0) register_command config diff hook 0x5617a23ec180
-165> 2018-01-12 13:08:36.904514 7f794f513e80 5 asok(0x5617a245c1c0) register_command config diff get hook 0x5617a23ec180
-164> 2018-01-12 13:08:36.904519 7f794f513e80 5 asok(0x5617a245c1c0) register_command log flush hook 0x5617a23ec180
-163> 2018-01-12 13:08:36.904523 7f794f513e80 5 asok(0x5617a245c1c0) register_command log dump hook 0x5617a23ec180
-162> 2018-01-12 13:08:36.904528 7f794f513e80 5 asok(0x5617a245c1c0) register_command log reopen hook 0x5617a23ec180
-161> 2018-01-12 13:08:36.904535 7f794f513e80 5 asok(0x5617a245c1c0) register_command dump_mempools hook 0x5617a24cde68
-160> 2018-01-12 13:08:36.912832 7f794f513e80 0 set uid:gid to 167:167 (ceph:ceph)
-159> 2018-01-12 13:08:36.912859 7f794f513e80 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 28726
-158> 2018-01-12 13:08:36.913016 7f794f513e80 0 pidfile_write: ignore empty --pid-file
-157> 2018-01-12 13:08:36.943733 7f794f513e80 5 asok(0x5617a245c1c0) init /var/run/ceph/ceph-mon.ceph001.asok
-156> 2018-01-12 13:08:36.943750 7f794f513e80 5 asok(0x5617a245c1c0) bind_and_listen /var/run/ceph/ceph-mon.ceph001.asok
-155> 2018-01-12 13:08:36.943845 7f794f513e80 5 asok(0x5617a245c1c0) register_command 0 hook 0x5617a23e8088
-154> 2018-01-12 13:08:36.943851 7f794f513e80 5 asok(0x5617a245c1c0) register_command version hook 0x5617a23e8088
-153> 2018-01-12 13:08:36.943855 7f794f513e80 5 asok(0x5617a245c1c0) register_command git_version hook 0x5617a23e8088
-152> 2018-01-12 13:08:36.943860 7f794f513e80 5 asok(0x5617a245c1c0) register_command help hook 0x5617a23ec1a0
-151> 2018-01-12 13:08:36.943863 7f794f513e80 5 asok(0x5617a245c1c0) register_command get_command_descriptions hook 0x5617a23ec1b0
-150> 2018-01-12 13:08:36.943906 7f794887d700 5 asok(0x5617a245c1c0) entry start
-149> 2018-01-12 13:08:36.951556 7f794f513e80 0 load: jerasure load: lrc load: isa
-148> 2018-01-12 13:08:36.951703 7f794f513e80 0 set rocksdb option compression = kNoCompression
-147> 2018-01-12 13:08:36.951716 7f794f513e80 0 set rocksdb option write_buffer_size = 33554432
-146> 2018-01-12 13:08:36.951742 7f794f513e80 0 set rocksdb option compression = kNoCompression
-145> 2018-01-12 13:08:36.951749 7f794f513e80 0 set rocksdb option write_buffer_size = 33554432
-144> 2018-01-12 13:08:36.951936 7f794f513e80 4 rocksdb: RocksDB version: 5.4.0
-143> 2018-01-12 13:08:36.951947 7f794f513e80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
-142> 2018-01-12 13:08:36.951951 7f794f513e80 4 rocksdb: Compile date Nov 30 2017
-141> 2018-01-12 13:08:36.951954 7f794f513e80 4 rocksdb: DB SUMMARY
-140> 2018-01-12 13:08:36.952011 7f794f513e80 4 rocksdb: CURRENT file: CURRENT
-139> 2018-01-12 13:08:36.952016 7f794f513e80 4 rocksdb: IDENTITY file: IDENTITY
-138> 2018-01-12 13:08:36.952020 7f794f513e80 4 rocksdb: MANIFEST file: MANIFEST-000064 size: 219 Bytes
-137> 2018-01-12 13:08:36.952023 7f794f513e80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph001/store.db dir, Total Num: 3, files: 000048.sst 000050.sst 000060.sst
-136> 2018-01-12 13:08:36.952025 7f794f513e80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph001/store.db: 000065.log size: 0 ;
-135> 2018-01-12 13:08:36.952028 7f794f513e80 4 rocksdb: Options.error_if_exists: 0
-134> 2018-01-12 13:08:36.952029 7f794f513e80 4 rocksdb: Options.create_if_missing: 0
-133> 2018-01-12 13:08:36.952031 7f794f513e80 4 rocksdb: Options.paranoid_checks: 1
-132> 2018-01-12 13:08:36.952032 7f794f513e80 4 rocksdb: Options.env: 0x5617a10fa040
-131> 2018-01-12 13:08:36.952033 7f794f513e80 4 rocksdb: Options.info_log: 0x5617a24ce1c0
-130> 2018-01-12 13:08:36.952034 7f794f513e80 4 rocksdb: Options.max_open_files: -1
-129> 2018-01-12 13:08:36.952035 7f794f513e80 4 rocksdb: Options.max_file_opening_threads: 16
-128> 2018-01-12 13:08:36.952035 7f794f513e80 4 rocksdb: Options.use_fsync: 0
-127> 2018-01-12 13:08:36.952037 7f794f513e80 4 rocksdb: Options.max_log_file_size: 0
-126> 2018-01-12 13:08:36.952038 7f794f513e80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-125> 2018-01-12 13:08:36.952039 7f794f513e80 4 rocksdb: Options.log_file_time_to_roll: 0
-124> 2018-01-12 13:08:36.952040 7f794f513e80 4 rocksdb: Options.keep_log_file_num: 1000
-123> 2018-01-12 13:08:36.952041 7f794f513e80 4 rocksdb: Options.recycle_log_file_num: 0
-122> 2018-01-12 13:08:36.952042 7f794f513e80 4 rocksdb: Options.allow_fallocate: 1
-121> 2018-01-12 13:08:36.952043 7f794f513e80 4 rocksdb: Options.allow_mmap_reads: 0
-120> 2018-01-12 13:08:36.952044 7f794f513e80 4 rocksdb: Options.allow_mmap_writes: 0
-119> 2018-01-12 13:08:36.952045 7f794f513e80 4 rocksdb: Options.use_direct_reads: 0
-118> 2018-01-12 13:08:36.952046 7f794f513e80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
-117> 2018-01-12 13:08:36.952047 7f794f513e80 4 rocksdb: Options.create_missing_column_families: 0
-116> 2018-01-12 13:08:36.952048 7f794f513e80 4 rocksdb: Options.db_log_dir:
-115> 2018-01-12 13:08:36.952049 7f794f513e80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-ceph001/store.db
-114> 2018-01-12 13:08:36.952050 7f794f513e80 4 rocksdb: Options.table_cache_numshardbits: 6
-113> 2018-01-12 13:08:36.952050 7f794f513e80 4 rocksdb: Options.max_subcompactions: 1
-112> 2018-01-12 13:08:36.952062 7f794f513e80 4 rocksdb: Options.max_background_flushes: 1
-111> 2018-01-12 13:08:36.952063 7f794f513e80 4 rocksdb: Options.WAL_ttl_seconds: 0
-110> 2018-01-12 13:08:36.952064 7f794f513e80 4 rocksdb: Options.WAL_size_limit_MB: 0
-109> 2018-01-12 13:08:36.952065 7f794f513e80 4 rocksdb: Options.manifest_preallocation_size: 4194304
-108> 2018-01-12 13:08:36.952066 7f794f513e80 4 rocksdb: Options.is_fd_close_on_exec: 1
-107> 2018-01-12 13:08:36.952067 7f794f513e80 4 rocksdb: Options.advise_random_on_open: 1
-106> 2018-01-12 13:08:36.952068 7f794f513e80 4 rocksdb: Options.db_write_buffer_size: 0
-105> 2018-01-12 13:08:36.952069 7f794f513e80 4 rocksdb: Options.access_hint_on_compaction_start: 1
-104> 2018-01-12 13:08:36.952069 7f794f513e80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-103> 2018-01-12 13:08:36.952070 7f794f513e80 4 rocksdb: Options.compaction_readahead_size: 0
-102> 2018-01-12 13:08:36.952071 7f794f513e80 4 rocksdb: Options.random_access_max_buffer_size: 1048576
-101> 2018-01-12 13:08:36.952072 7f794f513e80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
-100> 2018-01-12 13:08:36.952073 7f794f513e80 4 rocksdb: Options.use_adaptive_mutex: 0
-99> 2018-01-12 13:08:36.952074 7f794f513e80 4 rocksdb: Options.rate_limiter: (nil)
-98> 2018-01-12 13:08:36.952075 7f794f513e80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
-97> 2018-01-12 13:08:36.952076 7f794f513e80 4 rocksdb: Options.bytes_per_sync: 0
-96> 2018-01-12 13:08:36.952077 7f794f513e80 4 rocksdb: Options.wal_bytes_per_sync: 0
-95> 2018-01-12 13:08:36.952078 7f794f513e80 4 rocksdb: Options.wal_recovery_mode: 2
-94> 2018-01-12 13:08:36.952078 7f794f513e80 4 rocksdb: Options.enable_thread_tracking: 0
-93> 2018-01-12 13:08:36.952079 7f794f513e80 4 rocksdb: Options.allow_concurrent_memtable_write: 1
-92> 2018-01-12 13:08:36.952080 7f794f513e80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
-91> 2018-01-12 13:08:36.952081 7f794f513e80 4 rocksdb: Options.write_thread_max_yield_usec: 100
-90> 2018-01-12 13:08:36.952082 7f794f513e80 4 rocksdb: Options.write_thread_slow_yield_usec: 3
-89> 2018-01-12 13:08:36.952083 7f794f513e80 4 rocksdb: Options.row_cache: None
-88> 2018-01-12 13:08:36.952083 7f794f513e80 4 rocksdb: Options.wal_filter: None
-87> 2018-01-12 13:08:36.952084 7f794f513e80 4 rocksdb: Options.avoid_flush_during_recovery: 0
-86> 2018-01-12 13:08:36.952086 7f794f513e80 4 rocksdb: Options.base_background_compactions: 1
-85> 2018-01-12 13:08:36.952087 7f794f513e80 4 rocksdb: Options.max_background_compactions: 1
-84> 2018-01-12 13:08:36.952088 7f794f513e80 4 rocksdb: Options.avoid_flush_during_shutdown: 0
-83> 2018-01-12 13:08:36.952088 7f794f513e80 4 rocksdb: Options.delayed_write_rate : 16777216
-82> 2018-01-12 13:08:36.952089 7f794f513e80 4 rocksdb: Options.max_total_wal_size: 0
-81> 2018-01-12 13:08:36.952091 7f794f513e80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-80> 2018-01-12 13:08:36.952092 7f794f513e80 4 rocksdb: Options.stats_dump_period_sec: 600
-79> 2018-01-12 13:08:36.952093 7f794f513e80 4 rocksdb: Compression algorithms supported:
-78> 2018-01-12 13:08:36.952096 7f794f513e80 4 rocksdb: Snappy supported: 0
-77> 2018-01-12 13:08:36.952097 7f794f513e80 4 rocksdb: Zlib supported: 0
-76> 2018-01-12 13:08:36.952098 7f794f513e80 4 rocksdb: Bzip supported: 0
-75> 2018-01-12 13:08:36.952099 7f794f513e80 4 rocksdb: LZ4 supported: 0
-74> 2018-01-12 13:08:36.952100 7f794f513e80 4 rocksdb: ZSTD supported: 0
-73> 2018-01-12 13:08:36.952102 7f794f513e80 4 rocksdb: Fast CRC32 supported: 0
-72> 2018-01-12 13:08:36.952354 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000064
-71> 2018-01-12 13:08:36.952489 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
-70> 2018-01-12 13:08:36.952496 7f794f513e80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
-69> 2018-01-12 13:08:36.952498 7f794f513e80 4 rocksdb: Options.merge_operator:
-68> 2018-01-12 13:08:36.952502 7f794f513e80 4 rocksdb: Options.compaction_filter: None
-67> 2018-01-12 13:08:36.952503 7f794f513e80 4 rocksdb: Options.compaction_filter_factory: None
-66> 2018-01-12 13:08:36.952504 7f794f513e80 4 rocksdb: Options.memtable_factory: SkipListFactory
-65> 2018-01-12 13:08:36.952505 7f794f513e80 4 rocksdb: Options.table_factory: BlockBasedTable
-64> 2018-01-12 13:08:36.952525 7f794f513e80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617a23e80a0)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x5617a24068c8
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
-63> 2018-01-12 13:08:36.952547 7f794f513e80 4 rocksdb: Options.write_buffer_size: 33554432
-62> 2018-01-12 13:08:36.952548 7f794f513e80 4 rocksdb: Options.max_write_buffer_number: 2
-61> 2018-01-12 13:08:36.952550 7f794f513e80 4 rocksdb: Options.compression: NoCompression
-60> 2018-01-12 13:08:36.952561 7f794f513e80 4 rocksdb: Options.bottommost_compression: Disabled
-59> 2018-01-12 13:08:36.952563 7f794f513e80 4 rocksdb: Options.prefix_extractor: nullptr
-58> 2018-01-12 13:08:36.952564 7f794f513e80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
-57> 2018-01-12 13:08:36.952565 7f794f513e80 4 rocksdb: Options.num_levels: 7
-56> 2018-01-12 13:08:36.952565 7f794f513e80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
-55> 2018-01-12 13:08:36.952567 7f794f513e80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-54> 2018-01-12 13:08:36.952568 7f794f513e80 4 rocksdb: Options.compression_opts.window_bits: -14
-53> 2018-01-12 13:08:36.952568 7f794f513e80 4 rocksdb: Options.compression_opts.level: -1
-52> 2018-01-12 13:08:36.952569 7f794f513e80 4 rocksdb: Options.compression_opts.strategy: 0
-51> 2018-01-12 13:08:36.952570 7f794f513e80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
-50> 2018-01-12 13:08:36.952571 7f794f513e80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-49> 2018-01-12 13:08:36.952573 7f794f513e80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-48> 2018-01-12 13:08:36.952573 7f794f513e80 4 rocksdb: Options.level0_stop_writes_trigger: 36
-47> 2018-01-12 13:08:36.952574 7f794f513e80 4 rocksdb: Options.target_file_size_base: 67108864
-46> 2018-01-12 13:08:36.952575 7f794f513e80 4 rocksdb: Options.target_file_size_multiplier: 1
-45> 2018-01-12 13:08:36.952576 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_base: 268435456
-44> 2018-01-12 13:08:36.952577 7f794f513e80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-43> 2018-01-12 13:08:36.952578 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
-42> 2018-01-12 13:08:36.952583 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-41> 2018-01-12 13:08:36.952584 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-40> 2018-01-12 13:08:36.952585 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-39> 2018-01-12 13:08:36.952586 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-38> 2018-01-12 13:08:36.952588 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-37> 2018-01-12 13:08:36.952589 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-36> 2018-01-12 13:08:36.952589 7f794f513e80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-35> 2018-01-12 13:08:36.952590 7f794f513e80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-34> 2018-01-12 13:08:36.952591 7f794f513e80 4 rocksdb: Options.max_compaction_bytes: 1677721600
-33> 2018-01-12 13:08:36.952592 7f794f513e80 4 rocksdb: Options.arena_block_size: 4194304
-32> 2018-01-12 13:08:36.952593 7f794f513e80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
-31> 2018-01-12 13:08:36.952595 7f794f513e80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
-30> 2018-01-12 13:08:36.952595 7f794f513e80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
-29> 2018-01-12 13:08:36.952597 7f794f513e80 4 rocksdb: Options.disable_auto_compactions: 0
-28> 2018-01-12 13:08:36.952598 7f794f513e80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
-27> 2018-01-12 13:08:36.952599 7f794f513e80 4 rocksdb: Options.compaction_pri: kByCompensatedSize
-26> 2018-01-12 13:08:36.952600 7f794f513e80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-25> 2018-01-12 13:08:36.952601 7f794f513e80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-24> 2018-01-12 13:08:36.952602 7f794f513e80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-23> 2018-01-12 13:08:36.952603 7f794f513e80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-22> 2018-01-12 13:08:36.952604 7f794f513e80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-21> 2018-01-12 13:08:36.952605 7f794f513e80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-20> 2018-01-12 13:08:36.952616 7f794f513e80 4 rocksdb: Options.table_properties_collectors:
-19> 2018-01-12 13:08:36.952618 7f794f513e80 4 rocksdb: Options.inplace_update_support: 0
-18> 2018-01-12 13:08:36.952619 7f794f513e80 4 rocksdb: Options.inplace_update_num_locks: 10000
-17> 2018-01-12 13:08:36.952620 7f794f513e80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-16> 2018-01-12 13:08:36.952621 7f794f513e80 4 rocksdb: Options.memtable_huge_page_size: 0
-15> 2018-01-12 13:08:36.952622 7f794f513e80 4 rocksdb: Options.bloom_locality: 0
-14> 2018-01-12 13:08:36.952623 7f794f513e80 4 rocksdb: Options.max_successive_merges: 0
-13> 2018-01-12 13:08:36.952624 7f794f513e80 4 rocksdb: Options.optimize_filters_for_hits: 0
-12> 2018-01-12 13:08:36.952625 7f794f513e80 4 rocksdb: Options.paranoid_file_checks: 0
-11> 2018-01-12 13:08:36.952626 7f794f513e80 4 rocksdb: Options.force_consistency_checks: 0
-10> 2018-01-12 13:08:36.952627 7f794f513e80 4 rocksdb: Options.report_bg_io_stats: 0
-9> 2018-01-12 13:08:36.955246 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph001/store.db/MANIFEST-000064 succeeded,manifest_file_number is 64, next_file_number is 66, last_sequence is 10347, log_number is 0,prev_log_number is 0,max_column_family is 0
-8> 2018-01-12 13:08:36.955257 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 63
-7> 2018-01-12 13:08:36.955357 7f794f513e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916955344, "job": 1, "event": "recovery_started", "log_files": [65]}
-6> 2018-01-12 13:08:36.955365 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #65 mode 2
-5> 2018-01-12 13:08:36.955496 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 67
-4> 2018-01-12 13:08:36.956689 7f794f513e80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916956669, "job": 1, "event": "recovery_finished"}
-3> 2018-01-12 13:08:36.956943 7f794f513e80 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph001/store.db//MANIFEST-000064 type=3 #64 -- OK
-2> 2018-01-12 13:08:36.956970 7f794f513e80 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph001/store.db//000065.log type=0 #65 -- OK
-1> 2018-01-12 13:08:36.959779 7f794f513e80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5617a26a4000
0> 2018-01-12 13:08:36.964040 7f794f513e80 -1 *** Caught signal (Segmentation fault) **
in thread 7f794f513e80 thread_name:ceph-mon
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0x8eae11) [0x5617988d4e11]
2: (()+0xf5e0) [0x7f794e6765e0]
3: (malloc_usable_size()+0x8a) [0x7f794f0d8b7a]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x5617989a1eff]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x427) [0x5617989a3057]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5617989a32d6]
7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x5617989a3876]
8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x5617989a619f]
9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x2a4) [0x561798a635c4]
10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x561798969050]
11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x561798a10f04]
12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x561798a11489]
13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x561798a154a5]
14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x561798a1494a]
15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x561798491a42]
16: (main()+0xca7) [0x5617982bae37]
17: (__libc_start_main()+0xf5) [0x7f794baf8c05]
18: (()+0x378e1e) [0x561798362e1e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph001.log
--- end dump of recent events ---
2018-01-12 16:03:06.296305 7eff27bfe700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:388] [JOB 5] Try to delete WAL files size 262, prev total WAL file size 262, number of live WAL files 2.
2018-01-12 16:03:06.296416 7eff283ff700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:06.296430 7eff283ff700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:06.296451 7eff283ff700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:11.263456 7eff33265700 0 log_channel(cluster) log [INF] : mon.ceph001@0 won leader election with quorum 0,1
2018-01-12 16:03:11.265071 7eff33265700 0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 1/3 mons down, quorum ceph002,ceph003
2018-01-12 16:03:11.266543 7eff30a60700 0 log_channel(cluster) log [INF] : mon.1 10.141.8.181:6789/0
2018-01-12 16:03:11.275943 7eff2b3fe700 -1 *** Caught signal (Segmentation fault) **
in thread 7eff2b3fe700 thread_name:fn_monstore
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0x8eae11) [0x5566792b6e11]
2: (()+0xf5e0) [0x7eff3a73e5e0]
3: (malloc_usable_size()+0x8a) [0x7eff3b1a0b7a]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockB
asedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x556679383eff]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb:
:Block>*, bool)+0x427) [0x556679385057]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5566793852d6]
7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x556679385876]
8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x55667938819f]
9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+
0x2a4) [0x5566794455c4]
10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x55667934b050]
11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x5566793f2f04]
12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x5566793f3489]
13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x5566793f74a5]
14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x5566793f694a]
15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x556678e73a42]
16: (MonitorDBStore::get(std::string const&, std::string const&)+0x98) [0x556678d4a598]
17: (PaxosService::refresh(bool*)+0x5a) [0x556678ea1e4a]
18: (Monitor::refresh_from_paxos(bool*)+0x193) [0x556678d6f653]
19: (Paxos::do_refresh()+0x3f) [0x556678e8d71f]
20: (Paxos::commit_finish()+0x69f) [0x556678e976ff]
21: (C_Committed::finish(int)+0x31) [0x556678e9f6b1]
22: (Context::complete(int)+0x9) [0x556678d8f0c9]
23: (MonitorDBStore::C_DoTransaction::finish(int)+0xa7) [0x556678e9e7c7]
24: (Context::complete(int)+0x9) [0x556678d8f0c9]
25: (Finisher::finisher_thread_entry()+0x198) [0x556678fb52a8]
26: (()+0x7e25) [0x7eff3a736e25]
27: (clone()+0x6d) [0x7eff37c9734d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions