Project

General

Profile

Support #22680

mons segmentation faults New 12.2.2 cluster

Added by Kenneth Waegeman about 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Component(RADOS):
Monitor
Pull request ID:

Description

Hi all,

I installed a new Luminous 12.2.2 cluster. The monitors were up at first, but quickly started failing, segfaulting.

I only installed some mons, mgr, mds with ceph-deploy and osds with ceph volume. No pools or fs were created yet.

When I start all mons again, there is a short window i can see the cluster state:

[root@ceph001 ~]# ceph status
cluster:
id: 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
health: HEALTH_WARN
1/3 mons down, quorum ceph002,ceph003

services:
mon: 3 daemons, quorum ceph002,ceph003, out of quorum: ceph001
mgr: ceph001(active), standbys: ceph002, ceph003
osd: 7 osds: 4 up, 4 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 4223 MB used, 14899 GB / 14904 GB avail
pgs:

But this is only until I lose quorum again.

What could be the problem here?

Thanks!!

Kenneth


2018-01-12 13:08:36.912832 7f794f513e80  0 set uid:gid to 167:167 (ceph:ceph)
2018-01-12 13:08:36.912859 7f794f513e80  0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 28726
2018-01-12 13:08:36.913016 7f794f513e80  0 pidfile_write: ignore empty --pid-file
2018-01-12 13:08:36.951556 7f794f513e80  0 load: jerasure load: lrc load: isa
2018-01-12 13:08:36.951703 7f794f513e80  0  set rocksdb option compression = kNoCompression
2018-01-12 13:08:36.951716 7f794f513e80  0  set rocksdb option write_buffer_size = 33554432
2018-01-12 13:08:36.951742 7f794f513e80  0  set rocksdb option compression = kNoCompression
2018-01-12 13:08:36.951749 7f794f513e80  0  set rocksdb option write_buffer_size = 33554432
2018-01-12 13:08:36.951936 7f794f513e80  4 rocksdb: RocksDB version: 5.4.0

2018-01-12 13:08:36.951947 7f794f513e80  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-01-12 13:08:36.951951 7f794f513e80  4 rocksdb: Compile date Nov 30 2017
2018-01-12 13:08:36.951954 7f794f513e80  4 rocksdb: DB SUMMARY

2018-01-12 13:08:36.952011 7f794f513e80  4 rocksdb: CURRENT file: CURRENT

2018-01-12 13:08:36.952016 7f794f513e80  4 rocksdb: IDENTITY file:  IDENTITY

2018-01-12 13:08:36.952020 7f794f513e80  4 rocksdb: MANIFEST file:  MANIFEST-000064 size: 219 Bytes

2018-01-12 13:08:36.952023 7f794f513e80  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph001/store.db dir, Total Num: 3, files: 000048.sst 000050.sst 000060.sst

2018-01-12 13:08:36.952025 7f794f513e80  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph001/store.db: 000065.log size: 0 ;

2018-01-12 13:08:36.952028 7f794f513e80  4 rocksdb:                         Options.error_if_exists: 0
2018-01-12 13:08:36.952029 7f794f513e80  4 rocksdb:                       Options.create_if_missing: 0
2018-01-12 13:08:36.952031 7f794f513e80  4 rocksdb:                         Options.paranoid_checks: 1
2018-01-12 13:08:36.952032 7f794f513e80  4 rocksdb:                                     Options.env: 0x5617a10fa040
2018-01-12 13:08:36.952033 7f794f513e80  4 rocksdb:                                Options.info_log: 0x5617a24ce1c0
2018-01-12 13:08:36.952034 7f794f513e80  4 rocksdb:                          Options.max_open_files: -1
2018-01-12 13:08:36.952035 7f794f513e80  4 rocksdb: Options.max_file_opening_threads: 16
2018-01-12 13:08:36.952035 7f794f513e80  4 rocksdb:                               Options.use_fsync: 0
2018-01-12 13:08:36.952037 7f794f513e80  4 rocksdb:                       Options.max_log_file_size: 0
2018-01-12 13:08:36.952038 7f794f513e80  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
2018-01-12 13:08:36.952039 7f794f513e80  4 rocksdb:                   Options.log_file_time_to_roll: 0
2018-01-12 13:08:36.952040 7f794f513e80  4 rocksdb:                       Options.keep_log_file_num: 1000
2018-01-12 13:08:36.952041 7f794f513e80  4 rocksdb:                    Options.recycle_log_file_num: 0
2018-01-12 13:08:36.952042 7f794f513e80  4 rocksdb:                         Options.allow_fallocate: 1
2018-01-12 13:08:36.952043 7f794f513e80  4 rocksdb:                        Options.allow_mmap_reads: 0
2018-01-12 13:08:36.952044 7f794f513e80  4 rocksdb:                       Options.allow_mmap_writes: 0
2018-01-12 13:08:36.952045 7f794f513e80  4 rocksdb:                        Options.use_direct_reads: 0
2018-01-12 13:08:36.952046 7f794f513e80  4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2018-01-12 13:08:36.952047 7f794f513e80  4 rocksdb: Options.create_missing_column_families: 0
2018-01-12 13:08:36.952048 7f794f513e80  4 rocksdb:                              Options.db_log_dir:
2018-01-12 13:08:36.952049 7f794f513e80  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-ceph001/store.db
2018-01-12 13:08:36.952050 7f794f513e80  4 rocksdb: Options.table_cache_numshardbits: 6
2018-01-12 13:08:36.952050 7f794f513e80  4 rocksdb:                      Options.max_subcompactions: 1
2018-01-12 13:08:36.952062 7f794f513e80  4 rocksdb:                  Options.max_background_flushes: 1
2018-01-12 13:08:36.952063 7f794f513e80  4 rocksdb:                         Options.WAL_ttl_seconds: 0
2018-01-12 13:08:36.952064 7f794f513e80  4 rocksdb:                       Options.WAL_size_limit_MB: 0
2018-01-12 13:08:36.952065 7f794f513e80  4 rocksdb: Options.manifest_preallocation_size: 4194304
2018-01-12 13:08:36.952066 7f794f513e80  4 rocksdb:                     Options.is_fd_close_on_exec: 1
2018-01-12 13:08:36.952067 7f794f513e80  4 rocksdb:                   Options.advise_random_on_open: 1
2018-01-12 13:08:36.952068 7f794f513e80  4 rocksdb:                    Options.db_write_buffer_size: 0
2018-01-12 13:08:36.952069 7f794f513e80  4 rocksdb: Options.access_hint_on_compaction_start: 1
2018-01-12 13:08:36.952069 7f794f513e80  4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2018-01-12 13:08:36.952070 7f794f513e80  4 rocksdb: Options.compaction_readahead_size: 0
2018-01-12 13:08:36.952071 7f794f513e80  4 rocksdb: Options.random_access_max_buffer_size: 1048576
2018-01-12 13:08:36.952072 7f794f513e80  4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2018-01-12 13:08:36.952073 7f794f513e80  4 rocksdb:                      Options.use_adaptive_mutex: 0
2018-01-12 13:08:36.952074 7f794f513e80  4 rocksdb:                            Options.rate_limiter: (nil)
2018-01-12 13:08:36.952075 7f794f513e80  4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2018-01-12 13:08:36.952076 7f794f513e80  4 rocksdb:                          Options.bytes_per_sync: 0
2018-01-12 13:08:36.952077 7f794f513e80  4 rocksdb:                      Options.wal_bytes_per_sync: 0
2018-01-12 13:08:36.952078 7f794f513e80  4 rocksdb:                       Options.wal_recovery_mode: 2
2018-01-12 13:08:36.952078 7f794f513e80  4 rocksdb:                  Options.enable_thread_tracking: 0
2018-01-12 13:08:36.952079 7f794f513e80  4 rocksdb: Options.allow_concurrent_memtable_write: 1
2018-01-12 13:08:36.952080 7f794f513e80  4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2018-01-12 13:08:36.952081 7f794f513e80  4 rocksdb: Options.write_thread_max_yield_usec: 100
2018-01-12 13:08:36.952082 7f794f513e80  4 rocksdb: Options.write_thread_slow_yield_usec: 3
2018-01-12 13:08:36.952083 7f794f513e80  4 rocksdb:                               Options.row_cache: None
2018-01-12 13:08:36.952083 7f794f513e80  4 rocksdb:                              Options.wal_filter: None
2018-01-12 13:08:36.952084 7f794f513e80  4 rocksdb: Options.avoid_flush_during_recovery: 0
2018-01-12 13:08:36.952086 7f794f513e80  4 rocksdb: Options.base_background_compactions: 1
2018-01-12 13:08:36.952087 7f794f513e80  4 rocksdb: Options.max_background_compactions: 1
2018-01-12 13:08:36.952088 7f794f513e80  4 rocksdb: Options.avoid_flush_during_shutdown: 0
2018-01-12 13:08:36.952088 7f794f513e80  4 rocksdb: Options.delayed_write_rate : 16777216
2018-01-12 13:08:36.952089 7f794f513e80  4 rocksdb: Options.max_total_wal_size: 0
2018-01-12 13:08:36.952091 7f794f513e80  4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2018-01-12 13:08:36.952092 7f794f513e80  4 rocksdb:                   Options.stats_dump_period_sec: 600
2018-01-12 13:08:36.952093 7f794f513e80  4 rocksdb: Compression algorithms supported:
2018-01-12 13:08:36.952096 7f794f513e80  4 rocksdb:     Snappy supported: 0
2018-01-12 13:08:36.952097 7f794f513e80  4 rocksdb:     Zlib supported: 0
2018-01-12 13:08:36.952098 7f794f513e80  4 rocksdb:     Bzip supported: 0
2018-01-12 13:08:36.952099 7f794f513e80  4 rocksdb:     LZ4 supported: 0
2018-01-12 13:08:36.952100 7f794f513e80  4 rocksdb:     ZSTD supported: 0
2018-01-12 13:08:36.952102 7f794f513e80  4 rocksdb: Fast CRC32 supported: 0
2018-01-12 13:08:36.952354 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000064

2018-01-12 13:08:36.952489 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

2018-01-12 13:08:36.952496 7f794f513e80  4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2018-01-12 13:08:36.952498 7f794f513e80  4 rocksdb: Options.merge_operator:
2018-01-12 13:08:36.952502 7f794f513e80  4 rocksdb: Options.compaction_filter: None
2018-01-12 13:08:36.952503 7f794f513e80  4 rocksdb: Options.compaction_filter_factory: None
2018-01-12 13:08:36.952504 7f794f513e80  4 rocksdb: Options.memtable_factory: SkipListFactory
2018-01-12 13:08:36.952505 7f794f513e80  4 rocksdb: Options.table_factory: BlockBasedTable
2018-01-12 13:08:36.952525 7f794f513e80  4 rocksdb: table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617a23e80a0)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x5617a24068c8
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 134217728
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

2018-01-12 13:08:36.952547 7f794f513e80  4 rocksdb: Options.write_buffer_size: 33554432
2018-01-12 13:08:36.952548 7f794f513e80  4 rocksdb: Options.max_write_buffer_number: 2
2018-01-12 13:08:36.952550 7f794f513e80  4 rocksdb: Options.compression: NoCompression
2018-01-12 13:08:36.952561 7f794f513e80  4 rocksdb:                  Options.bottommost_compression: Disabled
2018-01-12 13:08:36.952563 7f794f513e80  4 rocksdb: Options.prefix_extractor: nullptr
2018-01-12 13:08:36.952564 7f794f513e80  4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-01-12 13:08:36.952565 7f794f513e80  4 rocksdb: Options.num_levels: 7
2018-01-12 13:08:36.952565 7f794f513e80  4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2018-01-12 13:08:36.952567 7f794f513e80  4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2018-01-12 13:08:36.952568 7f794f513e80  4 rocksdb: Options.compression_opts.window_bits: -14
2018-01-12 13:08:36.952568 7f794f513e80  4 rocksdb:                  Options.compression_opts.level: -1
2018-01-12 13:08:36.952569 7f794f513e80  4 rocksdb: Options.compression_opts.strategy: 0
2018-01-12 13:08:36.952570 7f794f513e80  4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2018-01-12 13:08:36.952571 7f794f513e80  4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2018-01-12 13:08:36.952573 7f794f513e80  4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2018-01-12 13:08:36.952573 7f794f513e80  4 rocksdb: Options.level0_stop_writes_trigger: 36
2018-01-12 13:08:36.952574 7f794f513e80  4 rocksdb:                   Options.target_file_size_base: 67108864
2018-01-12 13:08:36.952575 7f794f513e80  4 rocksdb: Options.target_file_size_multiplier: 1
2018-01-12 13:08:36.952576 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_base: 268435456
2018-01-12 13:08:36.952577 7f794f513e80  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2018-01-12 13:08:36.952578 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2018-01-12 13:08:36.952583 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-01-12 13:08:36.952584 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-01-12 13:08:36.952585 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-01-12 13:08:36.952586 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-01-12 13:08:36.952588 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-01-12 13:08:36.952589 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-01-12 13:08:36.952589 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-01-12 13:08:36.952590 7f794f513e80  4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2018-01-12 13:08:36.952591 7f794f513e80  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
2018-01-12 13:08:36.952592 7f794f513e80  4 rocksdb:                        Options.arena_block_size: 4194304
2018-01-12 13:08:36.952593 7f794f513e80  4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2018-01-12 13:08:36.952595 7f794f513e80  4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2018-01-12 13:08:36.952595 7f794f513e80  4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2018-01-12 13:08:36.952597 7f794f513e80  4 rocksdb: Options.disable_auto_compactions: 0
2018-01-12 13:08:36.952598 7f794f513e80  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
2018-01-12 13:08:36.952599 7f794f513e80  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
2018-01-12 13:08:36.952600 7f794f513e80  4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2018-01-12 13:08:36.952601 7f794f513e80  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2018-01-12 13:08:36.952602 7f794f513e80  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2018-01-12 13:08:36.952603 7f794f513e80  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2018-01-12 13:08:36.952604 7f794f513e80  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2018-01-12 13:08:36.952605 7f794f513e80  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-01-12 13:08:36.952616 7f794f513e80  4 rocksdb:                   Options.table_properties_collectors:
2018-01-12 13:08:36.952618 7f794f513e80  4 rocksdb:                   Options.inplace_update_support: 0
2018-01-12 13:08:36.952619 7f794f513e80  4 rocksdb:                 Options.inplace_update_num_locks: 10000
2018-01-12 13:08:36.952620 7f794f513e80  4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-01-12 13:08:36.952621 7f794f513e80  4 rocksdb: Options.memtable_huge_page_size: 0
2018-01-12 13:08:36.952622 7f794f513e80  4 rocksdb:                           Options.bloom_locality: 0
2018-01-12 13:08:36.952623 7f794f513e80  4 rocksdb:                    Options.max_successive_merges: 0
2018-01-12 13:08:36.952624 7f794f513e80  4 rocksdb: Options.optimize_filters_for_hits: 0
2018-01-12 13:08:36.952625 7f794f513e80  4 rocksdb: Options.paranoid_file_checks: 0
2018-01-12 13:08:36.952626 7f794f513e80  4 rocksdb: Options.force_consistency_checks: 0
2018-01-12 13:08:36.952627 7f794f513e80  4 rocksdb: Options.report_bg_io_stats: 0
2018-01-12 13:08:36.955246 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph001/store.db/MANIFEST-000064 succeeded,manifest_file_number is 64, next_file_number is 66, last_sequence is 10347, log_number is 0,prev_log_number is 0,max_column_family is 0

2018-01-12 13:08:36.955257 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 63

2018-01-12 13:08:36.955357 7f794f513e80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916955344, "job": 1, "event": "recovery_started", "log_files": [65]}
2018-01-12 13:08:36.955365 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #65 mode 2
2018-01-12 13:08:36.955496 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 67

2018-01-12 13:08:36.956689 7f794f513e80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916956669, "job": 1, "event": "recovery_finished"}
2018-01-12 13:08:36.959779 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5617a26a4000
2018-01-12 13:08:36.964040 7f794f513e80 -1 *** Caught signal (Segmentation fault) **
 in thread 7f794f513e80 thread_name:ceph-mon

 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
 1: (()+0x8eae11) [0x5617988d4e11]
 2: (()+0xf5e0) [0x7f794e6765e0]
 3: (malloc_usable_size()+0x8a) [0x7f794f0d8b7a]
 4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x5617989a1eff]
 5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x427) [0x5617989a3057]
 6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5617989a32d6]
 7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x5617989a3876]
 8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x5617989a619f]
 9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x2a4) [0x561798a635c4]
 10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x561798969050]
 11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x561798a10f04]
 12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x561798a11489]
 13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x561798a154a5]
 14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x561798a1494a]
 15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x561798491a42]
 16: (main()+0xca7) [0x5617982bae37]
 17: (__libc_start_main()+0xf5) [0x7f794baf8c05]
 18: (()+0x378e1e) [0x561798362e1e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -179> 2018-01-12 13:08:36.904404 7f794f513e80  5 asok(0x5617a245c1c0) register_command perfcounters_dump hook 0x5617a23ec180
  -178> 2018-01-12 13:08:36.904434 7f794f513e80  5 asok(0x5617a245c1c0) register_command 1 hook 0x5617a23ec180
  -177> 2018-01-12 13:08:36.904440 7f794f513e80  5 asok(0x5617a245c1c0) register_command perf dump hook 0x5617a23ec180
  -176> 2018-01-12 13:08:36.904442 7f794f513e80  5 asok(0x5617a245c1c0) register_command perfcounters_schema hook 0x5617a23ec180
  -175> 2018-01-12 13:08:36.904448 7f794f513e80  5 asok(0x5617a245c1c0) register_command perf histogram dump hook 0x5617a23ec180
  -174> 2018-01-12 13:08:36.904450 7f794f513e80  5 asok(0x5617a245c1c0) register_command 2 hook 0x5617a23ec180
  -173> 2018-01-12 13:08:36.904468 7f794f513e80  5 asok(0x5617a245c1c0) register_command perf schema hook 0x5617a23ec180
  -172> 2018-01-12 13:08:36.904473 7f794f513e80  5 asok(0x5617a245c1c0) register_command perf histogram schema hook 0x5617a23ec180
  -171> 2018-01-12 13:08:36.904485 7f794f513e80  5 asok(0x5617a245c1c0) register_command perf reset hook 0x5617a23ec180
  -170> 2018-01-12 13:08:36.904492 7f794f513e80  5 asok(0x5617a245c1c0) register_command config show hook 0x5617a23ec180
  -169> 2018-01-12 13:08:36.904496 7f794f513e80  5 asok(0x5617a245c1c0) register_command config help hook 0x5617a23ec180
  -168> 2018-01-12 13:08:36.904502 7f794f513e80  5 asok(0x5617a245c1c0) register_command config set hook 0x5617a23ec180
  -167> 2018-01-12 13:08:36.904505 7f794f513e80  5 asok(0x5617a245c1c0) register_command config get hook 0x5617a23ec180
  -166> 2018-01-12 13:08:36.904509 7f794f513e80  5 asok(0x5617a245c1c0) register_command config diff hook 0x5617a23ec180
  -165> 2018-01-12 13:08:36.904514 7f794f513e80  5 asok(0x5617a245c1c0) register_command config diff get hook 0x5617a23ec180
  -164> 2018-01-12 13:08:36.904519 7f794f513e80  5 asok(0x5617a245c1c0) register_command log flush hook 0x5617a23ec180
  -163> 2018-01-12 13:08:36.904523 7f794f513e80  5 asok(0x5617a245c1c0) register_command log dump hook 0x5617a23ec180
  -162> 2018-01-12 13:08:36.904528 7f794f513e80  5 asok(0x5617a245c1c0) register_command log reopen hook 0x5617a23ec180
  -161> 2018-01-12 13:08:36.904535 7f794f513e80  5 asok(0x5617a245c1c0) register_command dump_mempools hook 0x5617a24cde68
  -160> 2018-01-12 13:08:36.912832 7f794f513e80  0 set uid:gid to 167:167 (ceph:ceph)
  -159> 2018-01-12 13:08:36.912859 7f794f513e80  0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 28726
  -158> 2018-01-12 13:08:36.913016 7f794f513e80  0 pidfile_write: ignore empty --pid-file
  -157> 2018-01-12 13:08:36.943733 7f794f513e80  5 asok(0x5617a245c1c0) init /var/run/ceph/ceph-mon.ceph001.asok
  -156> 2018-01-12 13:08:36.943750 7f794f513e80  5 asok(0x5617a245c1c0) bind_and_listen /var/run/ceph/ceph-mon.ceph001.asok
  -155> 2018-01-12 13:08:36.943845 7f794f513e80  5 asok(0x5617a245c1c0) register_command 0 hook 0x5617a23e8088
  -154> 2018-01-12 13:08:36.943851 7f794f513e80  5 asok(0x5617a245c1c0) register_command version hook 0x5617a23e8088
  -153> 2018-01-12 13:08:36.943855 7f794f513e80  5 asok(0x5617a245c1c0) register_command git_version hook 0x5617a23e8088
  -152> 2018-01-12 13:08:36.943860 7f794f513e80  5 asok(0x5617a245c1c0) register_command help hook 0x5617a23ec1a0
  -151> 2018-01-12 13:08:36.943863 7f794f513e80  5 asok(0x5617a245c1c0) register_command get_command_descriptions hook 0x5617a23ec1b0
  -150> 2018-01-12 13:08:36.943906 7f794887d700  5 asok(0x5617a245c1c0) entry start
  -149> 2018-01-12 13:08:36.951556 7f794f513e80  0 load: jerasure load: lrc load: isa
  -148> 2018-01-12 13:08:36.951703 7f794f513e80  0  set rocksdb option compression = kNoCompression
  -147> 2018-01-12 13:08:36.951716 7f794f513e80  0  set rocksdb option write_buffer_size = 33554432
  -146> 2018-01-12 13:08:36.951742 7f794f513e80  0  set rocksdb option compression = kNoCompression
  -145> 2018-01-12 13:08:36.951749 7f794f513e80  0  set rocksdb option write_buffer_size = 33554432
  -144> 2018-01-12 13:08:36.951936 7f794f513e80  4 rocksdb: RocksDB version: 5.4.0

  -143> 2018-01-12 13:08:36.951947 7f794f513e80  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
  -142> 2018-01-12 13:08:36.951951 7f794f513e80  4 rocksdb: Compile date Nov 30 2017
  -141> 2018-01-12 13:08:36.951954 7f794f513e80  4 rocksdb: DB SUMMARY

  -140> 2018-01-12 13:08:36.952011 7f794f513e80  4 rocksdb: CURRENT file:  CURRENT

  -139> 2018-01-12 13:08:36.952016 7f794f513e80  4 rocksdb: IDENTITY file:  IDENTITY

  -138> 2018-01-12 13:08:36.952020 7f794f513e80  4 rocksdb: MANIFEST file:  MANIFEST-000064 size: 219 Bytes

  -137> 2018-01-12 13:08:36.952023 7f794f513e80  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph001/store.db dir, Total Num: 3, files: 000048.sst 000050.sst 000060.sst

  -136> 2018-01-12 13:08:36.952025 7f794f513e80  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph001/store.db: 000065.log size: 0 ;

  -135> 2018-01-12 13:08:36.952028 7f794f513e80  4 rocksdb:                         Options.error_if_exists: 0
  -134> 2018-01-12 13:08:36.952029 7f794f513e80  4 rocksdb:                       Options.create_if_missing: 0
  -133> 2018-01-12 13:08:36.952031 7f794f513e80  4 rocksdb:                         Options.paranoid_checks: 1
  -132> 2018-01-12 13:08:36.952032 7f794f513e80  4 rocksdb:                                     Options.env: 0x5617a10fa040
  -131> 2018-01-12 13:08:36.952033 7f794f513e80  4 rocksdb:                                Options.info_log: 0x5617a24ce1c0
  -130> 2018-01-12 13:08:36.952034 7f794f513e80  4 rocksdb:                          Options.max_open_files: -1
  -129> 2018-01-12 13:08:36.952035 7f794f513e80  4 rocksdb:                Options.max_file_opening_threads: 16
  -128> 2018-01-12 13:08:36.952035 7f794f513e80  4 rocksdb:                               Options.use_fsync: 0
  -127> 2018-01-12 13:08:36.952037 7f794f513e80  4 rocksdb:                       Options.max_log_file_size: 0
  -126> 2018-01-12 13:08:36.952038 7f794f513e80  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
  -125> 2018-01-12 13:08:36.952039 7f794f513e80  4 rocksdb:                   Options.log_file_time_to_roll: 0
  -124> 2018-01-12 13:08:36.952040 7f794f513e80  4 rocksdb:                       Options.keep_log_file_num: 1000
  -123> 2018-01-12 13:08:36.952041 7f794f513e80  4 rocksdb:                    Options.recycle_log_file_num: 0
  -122> 2018-01-12 13:08:36.952042 7f794f513e80  4 rocksdb:                         Options.allow_fallocate: 1
  -121> 2018-01-12 13:08:36.952043 7f794f513e80  4 rocksdb:                        Options.allow_mmap_reads: 0
  -120> 2018-01-12 13:08:36.952044 7f794f513e80  4 rocksdb:                       Options.allow_mmap_writes: 0
  -119> 2018-01-12 13:08:36.952045 7f794f513e80  4 rocksdb:                        Options.use_direct_reads: 0
  -118> 2018-01-12 13:08:36.952046 7f794f513e80  4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
  -117> 2018-01-12 13:08:36.952047 7f794f513e80  4 rocksdb:          Options.create_missing_column_families: 0
  -116> 2018-01-12 13:08:36.952048 7f794f513e80  4 rocksdb:                              Options.db_log_dir:
  -115> 2018-01-12 13:08:36.952049 7f794f513e80  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-ceph001/store.db
  -114> 2018-01-12 13:08:36.952050 7f794f513e80  4 rocksdb:                Options.table_cache_numshardbits: 6
  -113> 2018-01-12 13:08:36.952050 7f794f513e80  4 rocksdb:                      Options.max_subcompactions: 1
  -112> 2018-01-12 13:08:36.952062 7f794f513e80  4 rocksdb:                  Options.max_background_flushes: 1
  -111> 2018-01-12 13:08:36.952063 7f794f513e80  4 rocksdb:                         Options.WAL_ttl_seconds: 0
  -110> 2018-01-12 13:08:36.952064 7f794f513e80  4 rocksdb:                       Options.WAL_size_limit_MB: 0
  -109> 2018-01-12 13:08:36.952065 7f794f513e80  4 rocksdb:             Options.manifest_preallocation_size: 4194304
  -108> 2018-01-12 13:08:36.952066 7f794f513e80  4 rocksdb:                     Options.is_fd_close_on_exec: 1
  -107> 2018-01-12 13:08:36.952067 7f794f513e80  4 rocksdb:                   Options.advise_random_on_open: 1
  -106> 2018-01-12 13:08:36.952068 7f794f513e80  4 rocksdb:                    Options.db_write_buffer_size: 0
  -105> 2018-01-12 13:08:36.952069 7f794f513e80  4 rocksdb:         Options.access_hint_on_compaction_start: 1
  -104> 2018-01-12 13:08:36.952069 7f794f513e80  4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
  -103> 2018-01-12 13:08:36.952070 7f794f513e80  4 rocksdb:               Options.compaction_readahead_size: 0
  -102> 2018-01-12 13:08:36.952071 7f794f513e80  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
  -101> 2018-01-12 13:08:36.952072 7f794f513e80  4 rocksdb:           Options.writable_file_max_buffer_size: 1048576
  -100> 2018-01-12 13:08:36.952073 7f794f513e80  4 rocksdb:                      Options.use_adaptive_mutex: 0
   -99> 2018-01-12 13:08:36.952074 7f794f513e80  4 rocksdb:                            Options.rate_limiter: (nil)
   -98> 2018-01-12 13:08:36.952075 7f794f513e80  4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
   -97> 2018-01-12 13:08:36.952076 7f794f513e80  4 rocksdb:                          Options.bytes_per_sync: 0
   -96> 2018-01-12 13:08:36.952077 7f794f513e80  4 rocksdb:                      Options.wal_bytes_per_sync: 0
   -95> 2018-01-12 13:08:36.952078 7f794f513e80  4 rocksdb:                       Options.wal_recovery_mode: 2
   -94> 2018-01-12 13:08:36.952078 7f794f513e80  4 rocksdb:                  Options.enable_thread_tracking: 0
   -93> 2018-01-12 13:08:36.952079 7f794f513e80  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
   -92> 2018-01-12 13:08:36.952080 7f794f513e80  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
   -91> 2018-01-12 13:08:36.952081 7f794f513e80  4 rocksdb:             Options.write_thread_max_yield_usec: 100
   -90> 2018-01-12 13:08:36.952082 7f794f513e80  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
   -89> 2018-01-12 13:08:36.952083 7f794f513e80  4 rocksdb:                               Options.row_cache: None
   -88> 2018-01-12 13:08:36.952083 7f794f513e80  4 rocksdb:                              Options.wal_filter: None
   -87> 2018-01-12 13:08:36.952084 7f794f513e80  4 rocksdb:             Options.avoid_flush_during_recovery: 0
   -86> 2018-01-12 13:08:36.952086 7f794f513e80  4 rocksdb:             Options.base_background_compactions: 1
   -85> 2018-01-12 13:08:36.952087 7f794f513e80  4 rocksdb:             Options.max_background_compactions: 1
   -84> 2018-01-12 13:08:36.952088 7f794f513e80  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
   -83> 2018-01-12 13:08:36.952088 7f794f513e80  4 rocksdb:             Options.delayed_write_rate : 16777216
   -82> 2018-01-12 13:08:36.952089 7f794f513e80  4 rocksdb:             Options.max_total_wal_size: 0
   -81> 2018-01-12 13:08:36.952091 7f794f513e80  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
   -80> 2018-01-12 13:08:36.952092 7f794f513e80  4 rocksdb:                   Options.stats_dump_period_sec: 600
   -79> 2018-01-12 13:08:36.952093 7f794f513e80  4 rocksdb: Compression algorithms supported:
   -78> 2018-01-12 13:08:36.952096 7f794f513e80  4 rocksdb: Snappy supported: 0
   -77> 2018-01-12 13:08:36.952097 7f794f513e80  4 rocksdb: Zlib supported: 0
   -76> 2018-01-12 13:08:36.952098 7f794f513e80  4 rocksdb: Bzip supported: 0
   -75> 2018-01-12 13:08:36.952099 7f794f513e80  4 rocksdb: LZ4 supported: 0
   -74> 2018-01-12 13:08:36.952100 7f794f513e80  4 rocksdb: ZSTD supported: 0
   -73> 2018-01-12 13:08:36.952102 7f794f513e80  4 rocksdb: Fast CRC32 supported: 0
   -72> 2018-01-12 13:08:36.952354 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000064

   -71> 2018-01-12 13:08:36.952489 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

   -70> 2018-01-12 13:08:36.952496 7f794f513e80  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
   -69> 2018-01-12 13:08:36.952498 7f794f513e80  4 rocksdb:           Options.merge_operator:
   -68> 2018-01-12 13:08:36.952502 7f794f513e80  4 rocksdb:        Options.compaction_filter: None
   -67> 2018-01-12 13:08:36.952503 7f794f513e80  4 rocksdb:        Options.compaction_filter_factory: None
   -66> 2018-01-12 13:08:36.952504 7f794f513e80  4 rocksdb:         Options.memtable_factory: SkipListFactory
   -65> 2018-01-12 13:08:36.952505 7f794f513e80  4 rocksdb:            Options.table_factory: BlockBasedTable
   -64> 2018-01-12 13:08:36.952525 7f794f513e80  4 rocksdb:            table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617a23e80a0)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x5617a24068c8
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 134217728
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

   -63> 2018-01-12 13:08:36.952547 7f794f513e80  4 rocksdb:        Options.write_buffer_size: 33554432
   -62> 2018-01-12 13:08:36.952548 7f794f513e80  4 rocksdb: Options.max_write_buffer_number: 2
   -61> 2018-01-12 13:08:36.952550 7f794f513e80  4 rocksdb:          Options.compression: NoCompression
   -60> 2018-01-12 13:08:36.952561 7f794f513e80  4 rocksdb:                  Options.bottommost_compression: Disabled
   -59> 2018-01-12 13:08:36.952563 7f794f513e80  4 rocksdb:       Options.prefix_extractor: nullptr
   -58> 2018-01-12 13:08:36.952564 7f794f513e80  4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
   -57> 2018-01-12 13:08:36.952565 7f794f513e80  4 rocksdb:             Options.num_levels: 7
   -56> 2018-01-12 13:08:36.952565 7f794f513e80  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
   -55> 2018-01-12 13:08:36.952567 7f794f513e80  4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
   -54> 2018-01-12 13:08:36.952568 7f794f513e80  4 rocksdb:            Options.compression_opts.window_bits: -14
   -53> 2018-01-12 13:08:36.952568 7f794f513e80  4 rocksdb:                  Options.compression_opts.level: -1
   -52> 2018-01-12 13:08:36.952569 7f794f513e80  4 rocksdb:               Options.compression_opts.strategy: 0
   -51> 2018-01-12 13:08:36.952570 7f794f513e80  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
   -50> 2018-01-12 13:08:36.952571 7f794f513e80  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
   -49> 2018-01-12 13:08:36.952573 7f794f513e80  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
   -48> 2018-01-12 13:08:36.952573 7f794f513e80  4 rocksdb:              Options.level0_stop_writes_trigger: 36
   -47> 2018-01-12 13:08:36.952574 7f794f513e80  4 rocksdb:                   Options.target_file_size_base: 67108864
   -46> 2018-01-12 13:08:36.952575 7f794f513e80  4 rocksdb:             Options.target_file_size_multiplier: 1
   -45> 2018-01-12 13:08:36.952576 7f794f513e80  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
   -44> 2018-01-12 13:08:36.952577 7f794f513e80  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
   -43> 2018-01-12 13:08:36.952578 7f794f513e80  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
   -42> 2018-01-12 13:08:36.952583 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
   -41> 2018-01-12 13:08:36.952584 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
   -40> 2018-01-12 13:08:36.952585 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
   -39> 2018-01-12 13:08:36.952586 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
   -38> 2018-01-12 13:08:36.952588 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
   -37> 2018-01-12 13:08:36.952589 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
   -36> 2018-01-12 13:08:36.952589 7f794f513e80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
   -35> 2018-01-12 13:08:36.952590 7f794f513e80  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
   -34> 2018-01-12 13:08:36.952591 7f794f513e80  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
   -33> 2018-01-12 13:08:36.952592 7f794f513e80  4 rocksdb:                        Options.arena_block_size: 4194304
   -32> 2018-01-12 13:08:36.952593 7f794f513e80  4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
   -31> 2018-01-12 13:08:36.952595 7f794f513e80  4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
   -30> 2018-01-12 13:08:36.952595 7f794f513e80  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
   -29> 2018-01-12 13:08:36.952597 7f794f513e80  4 rocksdb:                Options.disable_auto_compactions: 0
   -28> 2018-01-12 13:08:36.952598 7f794f513e80  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
   -27> 2018-01-12 13:08:36.952599 7f794f513e80  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
   -26> 2018-01-12 13:08:36.952600 7f794f513e80  4 rocksdb: Options.compaction_options_universal.size_ratio: 1
   -25> 2018-01-12 13:08:36.952601 7f794f513e80  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
   -24> 2018-01-12 13:08:36.952602 7f794f513e80  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
   -23> 2018-01-12 13:08:36.952603 7f794f513e80  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
   -22> 2018-01-12 13:08:36.952604 7f794f513e80  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
   -21> 2018-01-12 13:08:36.952605 7f794f513e80  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
   -20> 2018-01-12 13:08:36.952616 7f794f513e80  4 rocksdb:                   Options.table_properties_collectors:
   -19> 2018-01-12 13:08:36.952618 7f794f513e80  4 rocksdb:                   Options.inplace_update_support: 0
   -18> 2018-01-12 13:08:36.952619 7f794f513e80  4 rocksdb:                 Options.inplace_update_num_locks: 10000
   -17> 2018-01-12 13:08:36.952620 7f794f513e80  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
   -16> 2018-01-12 13:08:36.952621 7f794f513e80  4 rocksdb: Options.memtable_huge_page_size: 0
   -15> 2018-01-12 13:08:36.952622 7f794f513e80  4 rocksdb:                           Options.bloom_locality: 0
   -14> 2018-01-12 13:08:36.952623 7f794f513e80  4 rocksdb:                    Options.max_successive_merges: 0
   -13> 2018-01-12 13:08:36.952624 7f794f513e80  4 rocksdb:                Options.optimize_filters_for_hits: 0
   -12> 2018-01-12 13:08:36.952625 7f794f513e80  4 rocksdb:                Options.paranoid_file_checks: 0
   -11> 2018-01-12 13:08:36.952626 7f794f513e80  4 rocksdb:                Options.force_consistency_checks: 0
   -10> 2018-01-12 13:08:36.952627 7f794f513e80  4 rocksdb:                Options.report_bg_io_stats: 0
    -9> 2018-01-12 13:08:36.955246 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph001/store.db/MANIFEST-000064 succeeded,manifest_file_number is 64, next_file_number is 66, last_sequence is 10347, log_number is 0,prev_log_number is 0,max_column_family is 0

    -8> 2018-01-12 13:08:36.955257 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 63

    -7> 2018-01-12 13:08:36.955357 7f794f513e80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916955344, "job": 1, "event": "recovery_started", "log_files": [65]}
    -6> 2018-01-12 13:08:36.955365 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #65 mode 2
    -5> 2018-01-12 13:08:36.955496 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 67

    -4> 2018-01-12 13:08:36.956689 7f794f513e80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1515758916956669, "job": 1, "event": "recovery_finished"}
    -3> 2018-01-12 13:08:36.956943 7f794f513e80  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph001/store.db//MANIFEST-000064 type=3 #64 -- OK

    -2> 2018-01-12 13:08:36.956970 7f794f513e80  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph001/store.db//000065.log type=0 #65 -- OK

    -1> 2018-01-12 13:08:36.959779 7f794f513e80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5617a26a4000
     0> 2018-01-12 13:08:36.964040 7f794f513e80 -1 *** Caught signal (Segmentation fault) **
 in thread 7f794f513e80 thread_name:ceph-mon

 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
 1: (()+0x8eae11) [0x5617988d4e11]
 2: (()+0xf5e0) [0x7f794e6765e0]
 3: (malloc_usable_size()+0x8a) [0x7f794f0d8b7a]
 4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x5617989a1eff]
 5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x427) [0x5617989a3057]
 6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5617989a32d6]
 7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x5617989a3876]
 8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x5617989a619f]
 9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x2a4) [0x561798a635c4]
 10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x561798969050]
 11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x561798a10f04]
 12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x561798a11489]
 13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x561798a154a5]
 14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x561798a1494a]
 15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x561798491a42]
 16: (main()+0xca7) [0x5617982bae37]
 17: (__libc_start_main()+0xf5) [0x7f794baf8c05]
 18: (()+0x378e1e) [0x561798362e1e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-mon.ceph001.log
--- end dump of recent events ---

2018-01-12 16:03:06.296305 7eff27bfe700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:388] [JOB 5] Try to delete WAL files size 262, prev total WAL file size 262, number of live WAL files 2.

2018-01-12 16:03:06.296416 7eff283ff700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:06.296430 7eff283ff700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:06.296451 7eff283ff700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/
BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
2018-01-12 16:03:11.263456 7eff33265700  0 log_channel(cluster) log [INF] : mon.ceph001@0 won leader election with quorum 0,1
2018-01-12 16:03:11.265071 7eff33265700  0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 1/3 mons down, quorum ceph002,ceph003
2018-01-12 16:03:11.266543 7eff30a60700  0 log_channel(cluster) log [INF] : mon.1 10.141.8.181:6789/0
2018-01-12 16:03:11.275943 7eff2b3fe700 -1 *** Caught signal (Segmentation fault) **
 in thread 7eff2b3fe700 thread_name:fn_monstore

 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
 1: (()+0x8eae11) [0x5566792b6e11]
 2: (()+0xf5e0) [0x7eff3a73e5e0]
 3: (malloc_usable_size()+0x8a) [0x7eff3b1a0b7a]
 4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockB
asedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x556679383eff]
 5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb:
:Block>*, bool)+0x427) [0x556679385057]
 6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x5566793852d6]
 7: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool)+0x96) [0x556679385876]
 8: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x51f) [0x55667938819f]
 9: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+
0x2a4) [0x5566794455c4]
 10: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x810) [0x55667934b050]
 11: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x5a4) [0x5566793f2f04]
 12: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x19) [0x5566793f3489]
 13: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x95) [0x5566793f74a5]
 14: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4a) [0x5566793f694a]
 15: (RocksDBStore::get(std::string const&, std::string const&, ceph::buffer::list*)+0x142) [0x556678e73a42]
 16: (MonitorDBStore::get(std::string const&, std::string const&)+0x98) [0x556678d4a598]
 17: (PaxosService::refresh(bool*)+0x5a) [0x556678ea1e4a]
 18: (Monitor::refresh_from_paxos(bool*)+0x193) [0x556678d6f653]
 19: (Paxos::do_refresh()+0x3f) [0x556678e8d71f]
 20: (Paxos::commit_finish()+0x69f) [0x556678e976ff]
 21: (C_Committed::finish(int)+0x31) [0x556678e9f6b1]
 22: (Context::complete(int)+0x9) [0x556678d8f0c9]
 23: (MonitorDBStore::C_DoTransaction::finish(int)+0xa7) [0x556678e9e7c7]
 24: (Context::complete(int)+0x9) [0x556678d8f0c9]
 25: (Finisher::finisher_thread_entry()+0x198) [0x556678fb52a8]
 26: (()+0x7e25) [0x7eff3a736e25]
 27: (clone()+0x6d) [0x7eff37c9734d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

History

#1 Updated by Greg Farnum about 6 years ago

  • Tracker changed from Bug to Support
  • Project changed from Ceph to RADOS
  • Category deleted (Monitor)
  • Component(RADOS) Monitor added

This is buried in the depths of RocksDB doing IO, so the only causes I know of/can think of are
1) you've found an unfixed bug in RocksDB
2) your disk is giving back bad data and RocksDB isn't handling that failure politely
3) you're using jemalloc against a binary built for tcmalloc?

#2 Updated by Kenneth Waegeman about 6 years ago

Thanks! We had jemalloc in LD_PRELOAD since Infernalis, so i didn't think about that. I removed this from sysconfig, and now the mons are stable again.

Thank you very much!

Kenneth

#3 Updated by Nathan Cutler about 6 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF