2018-07-21 21:20:44.512118 7f3d23173d80 0 set uid:gid to 167:167 (ceph:ceph) 2018-07-21 21:20:44.512138 7f3d23173d80 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 14561 2018-07-21 21:20:44.521794 7f3d23173d80 0 pidfile_write: ignore empty --pid-file 2018-07-21 21:20:44.562844 7f3d23173d80 0 load: jerasure load: lrc load: isa 2018-07-21 21:20:44.562930 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel 2018-07-21 21:20:44.562944 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block 2018-07-21 21:20:44.563244 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational 2018-07-21 21:20:44.563493 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _set_cache_sizes cache_size 209715200 meta 0.01 kv 0.99 data 0 2018-07-21 21:20:44.563520 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) close 2018-07-21 21:20:44.827802 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _mount path /var/lib/ceph/osd/ceph-24 2018-07-21 21:20:44.828308 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel 2018-07-21 21:20:44.828316 7f3d23173d80 1 bdev(0x55ad4e2b7600 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block 2018-07-21 21:20:44.828556 7f3d23173d80 1 bdev(0x55ad4e2b7600 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational 2018-07-21 21:20:44.828790 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _set_cache_sizes cache_size 209715200 meta 0.01 kv 0.99 data 0 2018-07-21 21:20:44.828870 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block.db type kernel 2018-07-21 21:20:44.828876 7f3d23173d80 1 bdev(0x55ad4e2b7a00 /var/lib/ceph/osd/ceph-24/block.db) open path /var/lib/ceph/osd/ceph-24/block.db 2018-07-21 21:20:44.829082 7f3d23173d80 1 bdev(0x55ad4e2b7a00 /var/lib/ceph/osd/ceph-24/block.db) open size 19999490048 (0x4a8100000, 19073 MB) block_size 4096 (4096 B) non-rotational 2018-07-21 21:20:44.829091 7f3d23173d80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-24/block.db size 19073 MB 2018-07-21 21:20:44.829317 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel 2018-07-21 21:20:44.829325 7f3d23173d80 1 bdev(0x55ad4e395a00 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block 2018-07-21 21:20:44.829500 7f3d23173d80 1 bdev(0x55ad4e395a00 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational 2018-07-21 21:20:44.829520 7f3d23173d80 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-24/block size 1787 GB 2018-07-21 21:20:44.829531 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block.wal type kernel 2018-07-21 21:20:44.829534 7f3d23173d80 1 bdev(0x55ad4e394e00 /var/lib/ceph/osd/ceph-24/block.wal) open path /var/lib/ceph/osd/ceph-24/block.wal 2018-07-21 21:20:44.829706 7f3d23173d80 1 bdev(0x55ad4e394e00 /var/lib/ceph/osd/ceph-24/block.wal) open size 1999634432 (0x77300000, 1907 MB) block_size 4096 (4096 B) non-rotational 2018-07-21 21:20:44.829718 7f3d23173d80 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-24/block.wal size 1907 MB 2018-07-21 21:20:44.829905 7f3d23173d80 1 bluefs mount 2018-07-21 21:20:44.838675 7f3d23173d80 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-21 21:20:44.838692 7f3d23173d80 0 set rocksdb option compression = kNoCompression 2018-07-21 21:20:44.838698 7f3d23173d80 0 set rocksdb option max_write_buffer_number = 4 2018-07-21 21:20:44.838702 7f3d23173d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-21 21:20:44.838706 7f3d23173d80 0 set rocksdb option recycle_log_file_num = 4 2018-07-21 21:20:44.838709 7f3d23173d80 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-21 21:20:44.838713 7f3d23173d80 0 set rocksdb option write_buffer_size = 268435456 2018-07-21 21:20:44.838742 7f3d23173d80 0 set rocksdb option compaction_readahead_size = 2097152 2018-07-21 21:20:44.838750 7f3d23173d80 0 set rocksdb option compression = kNoCompression 2018-07-21 21:20:44.838754 7f3d23173d80 0 set rocksdb option max_write_buffer_number = 4 2018-07-21 21:20:44.838758 7f3d23173d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2018-07-21 21:20:44.838761 7f3d23173d80 0 set rocksdb option recycle_log_file_num = 4 2018-07-21 21:20:44.838764 7f3d23173d80 0 set rocksdb option writable_file_max_buffer_size = 0 2018-07-21 21:20:44.838767 7f3d23173d80 0 set rocksdb option write_buffer_size = 268435456 2018-07-21 21:20:44.838949 7f3d23173d80 4 rocksdb: RocksDB version: 5.4.0 2018-07-21 21:20:44.838966 7f3d23173d80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-07-21 21:20:44.838968 7f3d23173d80 4 rocksdb: Compile date Jul 16 2018 2018-07-21 21:20:44.838979 7f3d23173d80 4 rocksdb: DB SUMMARY 2018-07-21 21:20:44.839008 7f3d23173d80 4 rocksdb: CURRENT file: CURRENT 2018-07-21 21:20:44.839012 7f3d23173d80 4 rocksdb: IDENTITY file: IDENTITY 2018-07-21 21:20:44.839018 7f3d23173d80 4 rocksdb: MANIFEST file: MANIFEST-000157 size: 350 Bytes 2018-07-21 21:20:44.839021 7f3d23173d80 4 rocksdb: SST files in db dir, Total Num: 6, files: 000004.sst 000010.sst 000013.sst 000019.sst 000021.sst 000024.sst 2018-07-21 21:20:44.839025 7f3d23173d80 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: 2018-07-21 21:20:44.839030 7f3d23173d80 4 rocksdb: Write Ahead Log file in db.wal: 000158.log size: 0 ; 2018-07-21 21:20:44.839033 7f3d23173d80 4 rocksdb: Options.error_if_exists: 0 2018-07-21 21:20:44.839041 7f3d23173d80 4 rocksdb: Options.create_if_missing: 0 2018-07-21 21:20:44.839043 7f3d23173d80 4 rocksdb: Options.paranoid_checks: 1 2018-07-21 21:20:44.839044 7f3d23173d80 4 rocksdb: Options.env: 0x55ad4e38d0a0 2018-07-21 21:20:44.839045 7f3d23173d80 4 rocksdb: Options.info_log: 0x55ad4e38d520 2018-07-21 21:20:44.839046 7f3d23173d80 4 rocksdb: Options.max_open_files: -1 2018-07-21 21:20:44.839047 7f3d23173d80 4 rocksdb: Options.max_file_opening_threads: 16 2018-07-21 21:20:44.839048 7f3d23173d80 4 rocksdb: Options.use_fsync: 0 2018-07-21 21:20:44.839049 7f3d23173d80 4 rocksdb: Options.max_log_file_size: 0 2018-07-21 21:20:44.839050 7f3d23173d80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2018-07-21 21:20:44.839051 7f3d23173d80 4 rocksdb: Options.log_file_time_to_roll: 0 2018-07-21 21:20:44.839052 7f3d23173d80 4 rocksdb: Options.keep_log_file_num: 1000 2018-07-21 21:20:44.839053 7f3d23173d80 4 rocksdb: Options.recycle_log_file_num: 4 2018-07-21 21:20:44.839054 7f3d23173d80 4 rocksdb: Options.allow_fallocate: 1 2018-07-21 21:20:44.839054 7f3d23173d80 4 rocksdb: Options.allow_mmap_reads: 0 2018-07-21 21:20:44.839055 7f3d23173d80 4 rocksdb: Options.allow_mmap_writes: 0 2018-07-21 21:20:44.839056 7f3d23173d80 4 rocksdb: Options.use_direct_reads: 0 2018-07-21 21:20:44.839057 7f3d23173d80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2018-07-21 21:20:44.839058 7f3d23173d80 4 rocksdb: Options.create_missing_column_families: 0 2018-07-21 21:20:44.839058 7f3d23173d80 4 rocksdb: Options.db_log_dir: 2018-07-21 21:20:44.839059 7f3d23173d80 4 rocksdb: Options.wal_dir: db.wal 2018-07-21 21:20:44.839060 7f3d23173d80 4 rocksdb: Options.table_cache_numshardbits: 6 2018-07-21 21:20:44.839061 7f3d23173d80 4 rocksdb: Options.max_subcompactions: 1 2018-07-21 21:20:44.839062 7f3d23173d80 4 rocksdb: Options.max_background_flushes: 1 2018-07-21 21:20:44.839062 7f3d23173d80 4 rocksdb: Options.WAL_ttl_seconds: 0 2018-07-21 21:20:44.839063 7f3d23173d80 4 rocksdb: Options.WAL_size_limit_MB: 0 2018-07-21 21:20:44.839064 7f3d23173d80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2018-07-21 21:20:44.839065 7f3d23173d80 4 rocksdb: Options.is_fd_close_on_exec: 1 2018-07-21 21:20:44.839066 7f3d23173d80 4 rocksdb: Options.advise_random_on_open: 1 2018-07-21 21:20:44.839066 7f3d23173d80 4 rocksdb: Options.db_write_buffer_size: 0 2018-07-21 21:20:44.839067 7f3d23173d80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2018-07-21 21:20:44.839068 7f3d23173d80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2018-07-21 21:20:44.839069 7f3d23173d80 4 rocksdb: Options.compaction_readahead_size: 2097152 2018-07-21 21:20:44.839069 7f3d23173d80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2018-07-21 21:20:44.839070 7f3d23173d80 4 rocksdb: Options.writable_file_max_buffer_size: 0 2018-07-21 21:20:44.839071 7f3d23173d80 4 rocksdb: Options.use_adaptive_mutex: 0 2018-07-21 21:20:44.839072 7f3d23173d80 4 rocksdb: Options.rate_limiter: (nil) 2018-07-21 21:20:44.839077 7f3d23173d80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2018-07-21 21:20:44.839079 7f3d23173d80 4 rocksdb: Options.bytes_per_sync: 0 2018-07-21 21:20:44.839079 7f3d23173d80 4 rocksdb: Options.wal_bytes_per_sync: 0 2018-07-21 21:20:44.839080 7f3d23173d80 4 rocksdb: Options.wal_recovery_mode: 0 2018-07-21 21:20:44.839082 7f3d23173d80 4 rocksdb: Options.enable_thread_tracking: 0 2018-07-21 21:20:44.839082 7f3d23173d80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2018-07-21 21:20:44.839083 7f3d23173d80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2018-07-21 21:20:44.839084 7f3d23173d80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2018-07-21 21:20:44.839085 7f3d23173d80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2018-07-21 21:20:44.839085 7f3d23173d80 4 rocksdb: Options.row_cache: None 2018-07-21 21:20:44.839086 7f3d23173d80 4 rocksdb: Options.wal_filter: None 2018-07-21 21:20:44.839087 7f3d23173d80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2018-07-21 21:20:44.839088 7f3d23173d80 4 rocksdb: Options.base_background_compactions: 1 2018-07-21 21:20:44.839089 7f3d23173d80 4 rocksdb: Options.max_background_compactions: 1 2018-07-21 21:20:44.839089 7f3d23173d80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2018-07-21 21:20:44.839090 7f3d23173d80 4 rocksdb: Options.delayed_write_rate : 16777216 2018-07-21 21:20:44.839091 7f3d23173d80 4 rocksdb: Options.max_total_wal_size: 0 2018-07-21 21:20:44.839092 7f3d23173d80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2018-07-21 21:20:44.839093 7f3d23173d80 4 rocksdb: Options.stats_dump_period_sec: 600 2018-07-21 21:20:44.839094 7f3d23173d80 4 rocksdb: Compression algorithms supported: 2018-07-21 21:20:44.839095 7f3d23173d80 4 rocksdb: Snappy supported: 1 2018-07-21 21:20:44.839096 7f3d23173d80 4 rocksdb: Zlib supported: 0 2018-07-21 21:20:44.839096 7f3d23173d80 4 rocksdb: Bzip supported: 0 2018-07-21 21:20:44.839097 7f3d23173d80 4 rocksdb: LZ4 supported: 0 2018-07-21 21:20:44.839098 7f3d23173d80 4 rocksdb: ZSTD supported: 0 2018-07-21 21:20:44.839099 7f3d23173d80 4 rocksdb: Fast CRC32 supported: 1 2018-07-21 21:20:44.839340 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000157 2018-07-21 21:20:44.839413 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2018-07-21 21:20:44.839423 7f3d23173d80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2018-07-21 21:20:44.839431 7f3d23173d80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2018-07-21 21:20:44.839433 7f3d23173d80 4 rocksdb: Options.compaction_filter: None 2018-07-21 21:20:44.839435 7f3d23173d80 4 rocksdb: Options.compaction_filter_factory: None 2018-07-21 21:20:44.839436 7f3d23173d80 4 rocksdb: Options.memtable_factory: SkipListFactory 2018-07-21 21:20:44.839439 7f3d23173d80 4 rocksdb: Options.table_factory: BlockBasedTable 2018-07-21 21:20:44.839460 7f3d23173d80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ad4e01e228) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55ad4e4900f8 block_cache_name: LRUCache block_cache_options: capacity : 207618048 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 2018-07-21 21:20:44.839465 7f3d23173d80 4 rocksdb: Options.write_buffer_size: 268435456 2018-07-21 21:20:44.839471 7f3d23173d80 4 rocksdb: Options.max_write_buffer_number: 4 2018-07-21 21:20:44.839473 7f3d23173d80 4 rocksdb: Options.compression: NoCompression 2018-07-21 21:20:44.839476 7f3d23173d80 4 rocksdb: Options.bottommost_compression: Disabled 2018-07-21 21:20:44.839478 7f3d23173d80 4 rocksdb: Options.prefix_extractor: nullptr 2018-07-21 21:20:44.839479 7f3d23173d80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2018-07-21 21:20:44.839481 7f3d23173d80 4 rocksdb: Options.num_levels: 7 2018-07-21 21:20:44.839482 7f3d23173d80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2018-07-21 21:20:44.839483 7f3d23173d80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2018-07-21 21:20:44.839485 7f3d23173d80 4 rocksdb: Options.compression_opts.window_bits: -14 2018-07-21 21:20:44.839486 7f3d23173d80 4 rocksdb: Options.compression_opts.level: -1 2018-07-21 21:20:44.839488 7f3d23173d80 4 rocksdb: Options.compression_opts.strategy: 0 2018-07-21 21:20:44.839489 7f3d23173d80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2018-07-21 21:20:44.839491 7f3d23173d80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2018-07-21 21:20:44.839492 7f3d23173d80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2018-07-21 21:20:44.839493 7f3d23173d80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2018-07-21 21:20:44.839494 7f3d23173d80 4 rocksdb: Options.target_file_size_base: 67108864 2018-07-21 21:20:44.839496 7f3d23173d80 4 rocksdb: Options.target_file_size_multiplier: 1 2018-07-21 21:20:44.839497 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2018-07-21 21:20:44.839498 7f3d23173d80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2018-07-21 21:20:44.839500 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2018-07-21 21:20:44.839503 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2018-07-21 21:20:44.839505 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2018-07-21 21:20:44.839507 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2018-07-21 21:20:44.839508 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2018-07-21 21:20:44.839509 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2018-07-21 21:20:44.839511 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2018-07-21 21:20:44.839512 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2018-07-21 21:20:44.839513 7f3d23173d80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2018-07-21 21:20:44.839515 7f3d23173d80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2018-07-21 21:20:44.839517 7f3d23173d80 4 rocksdb: Options.arena_block_size: 33554432 2018-07-21 21:20:44.839518 7f3d23173d80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2018-07-21 21:20:44.839519 7f3d23173d80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2018-07-21 21:20:44.839521 7f3d23173d80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2018-07-21 21:20:44.839522 7f3d23173d80 4 rocksdb: Options.disable_auto_compactions: 0 2018-07-21 21:20:44.839524 7f3d23173d80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2018-07-21 21:20:44.839526 7f3d23173d80 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2018-07-21 21:20:44.839527 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2018-07-21 21:20:44.839529 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2018-07-21 21:20:44.839530 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2018-07-21 21:20:44.839532 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2018-07-21 21:20:44.839533 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2018-07-21 21:20:44.839535 7f3d23173d80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2018-07-21 21:20:44.839536 7f3d23173d80 4 rocksdb: Options.table_properties_collectors: 2018-07-21 21:20:44.839537 7f3d23173d80 4 rocksdb: Options.inplace_update_support: 0 2018-07-21 21:20:44.839539 7f3d23173d80 4 rocksdb: Options.inplace_update_num_locks: 10000 2018-07-21 21:20:44.839540 7f3d23173d80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2018-07-21 21:20:44.839542 7f3d23173d80 4 rocksdb: Options.memtable_huge_page_size: 0 2018-07-21 21:20:44.839544 7f3d23173d80 4 rocksdb: Options.bloom_locality: 0 2018-07-21 21:20:44.839545 7f3d23173d80 4 rocksdb: Options.max_successive_merges: 0 2018-07-21 21:20:44.839546 7f3d23173d80 4 rocksdb: Options.optimize_filters_for_hits: 0 2018-07-21 21:20:44.839547 7f3d23173d80 4 rocksdb: Options.paranoid_file_checks: 0 2018-07-21 21:20:44.839548 7f3d23173d80 4 rocksdb: Options.force_consistency_checks: 0 2018-07-21 21:20:44.839549 7f3d23173d80 4 rocksdb: Options.report_bg_io_stats: 0 2018-07-21 21:20:44.841488 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000157 succeeded,manifest_file_number is 157, next_file_number is 159, last_sequence is 253029, log_number is 0,prev_log_number is 0,max_column_family is 0 2018-07-21 21:20:44.841501 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 156 2018-07-21 21:20:44.841556 7f3d23173d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844841547, "job": 1, "event": "recovery_started", "log_files": [158]} 2018-07-21 21:20:44.841563 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:482] Recovering log #158 mode 0 2018-07-21 21:20:44.841577 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2395] Creating manifest 160 2018-07-21 21:20:44.842391 7f3d23173d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844842388, "job": 1, "event": "recovery_finished"} 2018-07-21 21:20:44.845424 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55ad4e2c2b00 2018-07-21 21:20:44.845488 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2018-07-21 21:20:44.845760 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- 2018-07-21 21:20:44.845790 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:448] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 6/0 4.77 MB 1.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 6/0 4.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 6/0 4.77 MB 1.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 6/0 4.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2018-07-21 21:20:44.845926 7f3d0e782700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 6@0 files to L1, score 1.50 2018-07-21 21:20:44.845954 7f3d0e782700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 0, inputs: [24(1406B) 21(14KB) 19(4561KB) 13(308KB) 10(1391B) 4(1456B)] 2018-07-21 21:20:44.846000 7f3d0e782700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844845965, "job": 3, "event": "compaction_started", "files_L0": [24, 21, 19, 13, 10, 4], "score": 1.5, "input_data_size": 5006316} 2018-07-21 21:20:44.846921 7f3d23173d80 1 freelist init 2018-07-21 21:20:44.847517 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_alloc opening allocation metadata 2018-07-21 21:20:44.847652 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_alloc loaded 1787 G in 1 extents 2018-07-21 21:20:44.854250 7f3d23173d80 0 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2018-07-21 21:20:44.854289 7f3d23173d80 0 _get_class not permitted to load lua 2018-07-21 21:20:44.854303 7f3d23173d80 0 _get_class not permitted to load kvs 2018-07-21 21:20:44.859996 7f3d23173d80 0 _get_class not permitted to load sdk 2018-07-21 21:20:44.860460 7f3d23173d80 0 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello 2018-07-21 21:20:44.864106 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1591] Compaction error: Corruption: block checksum mismatch 2018-07-21 21:20:44.864130 7f3d0e782700 4 rocksdb: (Original Log Time 2018/07/21-21:20:44.864026) [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[6 0 0 0 0 0 0] max score 0.00, MB/sec: 278.4 rd, 20.3 wr, level 1, files in(6, 0) out(1) MB in(4.8, 0.0) out(0.3), read-write-amplify(1.1) write-amplify(0.1) Corruption: block checksum mismatch, records in: 4782 2018-07-21 21:20:44.864140 7f3d0e782700 4 rocksdb: (Original Log Time 2018/07/21-21:20:44.864089) EVENT_LOG_v1 {"time_micros": 1532200844864063, "job": 3, "event": "compaction_finished", "compaction_time_micros": 17983, "output_level": 1, "num_output_files": 1, "total_output_size": 365633, "num_input_records": 3539, "num_output_records": 1817, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 0, 0, 0, 0, 0, 0]} 2018-07-21 21:20:44.864146 7f3d0e782700 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1275] Waiting after background compaction error: Corruption: block checksum mismatch, Accumulated background error counts: 1 2018-07-21 21:20:44.865208 7f3d23173d80 0 osd.24 763 crush map has features 288514051259236352, adjusting msgr requires for clients 2018-07-21 21:20:44.865220 7f3d23173d80 0 osd.24 763 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons 2018-07-21 21:20:44.865226 7f3d23173d80 0 osd.24 763 crush map has features 1009089991638532096, adjusting msgr requires for osds 2018-07-21 21:20:44.873029 7f3d23173d80 0 osd.24 763 load_pgs 2018-07-21 21:20:44.907849 7f3d0ff85700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = O key = 0x7f80000000000000097000000021213dfffffffffffffffeffffffffffffffff'o' Value size = 29) Put( Prefix = S key = 'nid_max' Value size = 8) Put( Prefix = S key = 'blobid_max' Value size = 8) 2018-07-21 21:20:44.911270 7f3d0ff85700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f3d0ff85700 time 2018-07-21 21:20:44.907898 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueStore.cc: 8526: FAILED assert(r == 0) ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55ad44902850] 2: (BlueStore::_kv_sync_thread()+0x3472) [0x55ad447b1042] 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ad447f792d] 4: (()+0x7e25) [0x7f3d20845e25] 5: (clone()+0x6d) [0x7f3d1f936bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -275> 2018-07-21 21:20:44.503577 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perfcounters_dump hook 0x55ad4e020190 -274> 2018-07-21 21:20:44.503600 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command 1 hook 0x55ad4e020190 -273> 2018-07-21 21:20:44.503609 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perf dump hook 0x55ad4e020190 -272> 2018-07-21 21:20:44.503613 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perfcounters_schema hook 0x55ad4e020190 -271> 2018-07-21 21:20:44.503619 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perf histogram dump hook 0x55ad4e020190 -270> 2018-07-21 21:20:44.503624 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command 2 hook 0x55ad4e020190 -269> 2018-07-21 21:20:44.503628 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perf schema hook 0x55ad4e020190 -268> 2018-07-21 21:20:44.503632 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perf histogram schema hook 0x55ad4e020190 -267> 2018-07-21 21:20:44.503643 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command perf reset hook 0x55ad4e020190 -266> 2018-07-21 21:20:44.503648 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config show hook 0x55ad4e020190 -265> 2018-07-21 21:20:44.503654 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config help hook 0x55ad4e020190 -264> 2018-07-21 21:20:44.503658 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config set hook 0x55ad4e020190 -263> 2018-07-21 21:20:44.503663 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config get hook 0x55ad4e020190 -262> 2018-07-21 21:20:44.503668 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config diff hook 0x55ad4e020190 -261> 2018-07-21 21:20:44.503672 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command config diff get hook 0x55ad4e020190 -260> 2018-07-21 21:20:44.503677 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command log flush hook 0x55ad4e020190 -259> 2018-07-21 21:20:44.503683 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command log dump hook 0x55ad4e020190 -258> 2018-07-21 21:20:44.503686 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command log reopen hook 0x55ad4e020190 -257> 2018-07-21 21:20:44.503699 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command dump_mempools hook 0x55ad4e217228 -256> 2018-07-21 21:20:44.512118 7f3d23173d80 0 set uid:gid to 167:167 (ceph:ceph) -255> 2018-07-21 21:20:44.512138 7f3d23173d80 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 14561 -254> 2018-07-21 21:20:44.512180 7f3d23173d80 5 object store type is bluestore -253> 2018-07-21 21:20:44.517652 7f3d1df64700 2 Event(0x55ad4e2b6480 nevent=5000 time_id=1).set_owner idx=0 owner=139900472411904 -252> 2018-07-21 21:20:44.517710 7f3d1d763700 2 Event(0x55ad4e2b6880 nevent=5000 time_id=1).set_owner idx=1 owner=139900464019200 -251> 2018-07-21 21:20:44.517742 7f3d1cf62700 2 Event(0x55ad4e2b6c80 nevent=5000 time_id=1).set_owner idx=2 owner=139900455626496 -250> 2018-07-21 21:20:44.518865 7f3d23173d80 1 -- 100.64.252.8:0/0 learned_addr learned my addr 100.64.252.8:0/0 -249> 2018-07-21 21:20:44.518901 7f3d23173d80 1 -- 100.64.252.8:6815/14561 _finish_bind bind my_inst.addr is 100.64.252.8:6815/14561 -248> 2018-07-21 21:20:44.519641 7f3d23173d80 1 -- 100.64.252.8:0/0 learned_addr learned my addr 100.64.252.8:0/0 -247> 2018-07-21 21:20:44.519650 7f3d23173d80 1 -- 100.64.252.8:6831/14561 _finish_bind bind my_inst.addr is 100.64.252.8:6831/14561 -246> 2018-07-21 21:20:44.520632 7f3d23173d80 1 -- 100.64.252.8:0/0 learned_addr learned my addr 100.64.252.8:0/0 -245> 2018-07-21 21:20:44.520640 7f3d23173d80 1 -- 100.64.252.8:6843/14561 _finish_bind bind my_inst.addr is 100.64.252.8:6843/14561 -244> 2018-07-21 21:20:44.521781 7f3d23173d80 1 -- 100.64.252.8:0/0 learned_addr learned my addr 100.64.252.8:0/0 -243> 2018-07-21 21:20:44.521788 7f3d23173d80 1 -- 100.64.252.8:6851/14561 _finish_bind bind my_inst.addr is 100.64.252.8:6851/14561 -242> 2018-07-21 21:20:44.521794 7f3d23173d80 0 pidfile_write: ignore empty --pid-file -241> 2018-07-21 21:20:44.554613 7f3d23173d80 5 asok(0x55ad4e0581c0) init /var/run/ceph/ceph-osd.24.asok -240> 2018-07-21 21:20:44.554640 7f3d23173d80 5 asok(0x55ad4e0581c0) bind_and_listen /var/run/ceph/ceph-osd.24.asok -239> 2018-07-21 21:20:44.554719 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command 0 hook 0x55ad4e01e1d8 -238> 2018-07-21 21:20:44.554727 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command version hook 0x55ad4e01e1d8 -237> 2018-07-21 21:20:44.554731 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command git_version hook 0x55ad4e01e1d8 -236> 2018-07-21 21:20:44.554734 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command help hook 0x55ad4e020590 -235> 2018-07-21 21:20:44.554736 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command get_command_descriptions hook 0x55ad4e0205a0 -234> 2018-07-21 21:20:44.554836 7f3d23173d80 10 monclient: build_initial_monmap -233> 2018-07-21 21:20:44.554864 7f3d1b7e4700 5 asok(0x55ad4e0581c0) entry start -232> 2018-07-21 21:20:44.562844 7f3d23173d80 0 load: jerasure load: lrc load: isa -231> 2018-07-21 21:20:44.562884 7f3d23173d80 5 adding auth protocol: cephx -230> 2018-07-21 21:20:44.562888 7f3d23173d80 5 adding auth protocol: cephx -229> 2018-07-21 21:20:44.562930 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel -228> 2018-07-21 21:20:44.562944 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block -227> 2018-07-21 21:20:44.563244 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational -226> 2018-07-21 21:20:44.563493 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _set_cache_sizes cache_size 209715200 meta 0.01 kv 0.99 data 0 -225> 2018-07-21 21:20:44.563520 7f3d23173d80 1 bdev(0x55ad4e392c00 /var/lib/ceph/osd/ceph-24/block) close -224> 2018-07-21 21:20:44.827449 7f3d23173d80 5 asok(0x55ad4e0581c0) register_command objecter_requests hook 0x55ad4e020620 -223> 2018-07-21 21:20:44.827549 7f3d23173d80 1 -- 100.64.252.8:6815/14561 start start -222> 2018-07-21 21:20:44.827563 7f3d23173d80 1 -- - start start -221> 2018-07-21 21:20:44.827565 7f3d23173d80 1 -- - start start -220> 2018-07-21 21:20:44.827566 7f3d23173d80 1 -- 100.64.252.8:6851/14561 start start -219> 2018-07-21 21:20:44.827570 7f3d23173d80 1 -- 100.64.252.8:6843/14561 start start -218> 2018-07-21 21:20:44.827572 7f3d23173d80 1 -- 100.64.252.8:6831/14561 start start -217> 2018-07-21 21:20:44.827575 7f3d23173d80 1 -- - start start -216> 2018-07-21 21:20:44.827769 7f3d23173d80 2 osd.24 0 init /var/lib/ceph/osd/ceph-24 (looks like hdd) -215> 2018-07-21 21:20:44.827779 7f3d23173d80 2 osd.24 0 journal /var/lib/ceph/osd/ceph-24/journal -214> 2018-07-21 21:20:44.827802 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _mount path /var/lib/ceph/osd/ceph-24 -213> 2018-07-21 21:20:44.828308 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel -212> 2018-07-21 21:20:44.828316 7f3d23173d80 1 bdev(0x55ad4e2b7600 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block -211> 2018-07-21 21:20:44.828556 7f3d23173d80 1 bdev(0x55ad4e2b7600 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational -210> 2018-07-21 21:20:44.828790 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _set_cache_sizes cache_size 209715200 meta 0.01 kv 0.99 data 0 -209> 2018-07-21 21:20:44.828870 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block.db type kernel -208> 2018-07-21 21:20:44.828876 7f3d23173d80 1 bdev(0x55ad4e2b7a00 /var/lib/ceph/osd/ceph-24/block.db) open path /var/lib/ceph/osd/ceph-24/block.db -207> 2018-07-21 21:20:44.829082 7f3d23173d80 1 bdev(0x55ad4e2b7a00 /var/lib/ceph/osd/ceph-24/block.db) open size 19999490048 (0x4a8100000, 19073 MB) block_size 4096 (4096 B) non-rotational -206> 2018-07-21 21:20:44.829091 7f3d23173d80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-24/block.db size 19073 MB -205> 2018-07-21 21:20:44.829317 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block type kernel -204> 2018-07-21 21:20:44.829325 7f3d23173d80 1 bdev(0x55ad4e395a00 /var/lib/ceph/osd/ceph-24/block) open path /var/lib/ceph/osd/ceph-24/block -203> 2018-07-21 21:20:44.829500 7f3d23173d80 1 bdev(0x55ad4e395a00 /var/lib/ceph/osd/ceph-24/block) open size 1919711969280 (0x1bef7c00000, 1787 GB) block_size 4096 (4096 B) rotational -202> 2018-07-21 21:20:44.829520 7f3d23173d80 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-24/block size 1787 GB -201> 2018-07-21 21:20:44.829531 7f3d23173d80 1 bdev create path /var/lib/ceph/osd/ceph-24/block.wal type kernel -200> 2018-07-21 21:20:44.829534 7f3d23173d80 1 bdev(0x55ad4e394e00 /var/lib/ceph/osd/ceph-24/block.wal) open path /var/lib/ceph/osd/ceph-24/block.wal -199> 2018-07-21 21:20:44.829706 7f3d23173d80 1 bdev(0x55ad4e394e00 /var/lib/ceph/osd/ceph-24/block.wal) open size 1999634432 (0x77300000, 1907 MB) block_size 4096 (4096 B) non-rotational -198> 2018-07-21 21:20:44.829718 7f3d23173d80 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-24/block.wal size 1907 MB -197> 2018-07-21 21:20:44.829905 7f3d23173d80 1 bluefs mount -196> 2018-07-21 21:20:44.838675 7f3d23173d80 0 set rocksdb option compaction_readahead_size = 2097152 -195> 2018-07-21 21:20:44.838692 7f3d23173d80 0 set rocksdb option compression = kNoCompression -194> 2018-07-21 21:20:44.838698 7f3d23173d80 0 set rocksdb option max_write_buffer_number = 4 -193> 2018-07-21 21:20:44.838702 7f3d23173d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 -192> 2018-07-21 21:20:44.838706 7f3d23173d80 0 set rocksdb option recycle_log_file_num = 4 -191> 2018-07-21 21:20:44.838709 7f3d23173d80 0 set rocksdb option writable_file_max_buffer_size = 0 -190> 2018-07-21 21:20:44.838713 7f3d23173d80 0 set rocksdb option write_buffer_size = 268435456 -189> 2018-07-21 21:20:44.838742 7f3d23173d80 0 set rocksdb option compaction_readahead_size = 2097152 -188> 2018-07-21 21:20:44.838750 7f3d23173d80 0 set rocksdb option compression = kNoCompression -187> 2018-07-21 21:20:44.838754 7f3d23173d80 0 set rocksdb option max_write_buffer_number = 4 -186> 2018-07-21 21:20:44.838758 7f3d23173d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 -185> 2018-07-21 21:20:44.838761 7f3d23173d80 0 set rocksdb option recycle_log_file_num = 4 -184> 2018-07-21 21:20:44.838764 7f3d23173d80 0 set rocksdb option writable_file_max_buffer_size = 0 -183> 2018-07-21 21:20:44.838767 7f3d23173d80 0 set rocksdb option write_buffer_size = 268435456 -182> 2018-07-21 21:20:44.838949 7f3d23173d80 4 rocksdb: RocksDB version: 5.4.0 -181> 2018-07-21 21:20:44.838966 7f3d23173d80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -180> 2018-07-21 21:20:44.838968 7f3d23173d80 4 rocksdb: Compile date Jul 16 2018 -179> 2018-07-21 21:20:44.838979 7f3d23173d80 4 rocksdb: DB SUMMARY -178> 2018-07-21 21:20:44.839008 7f3d23173d80 4 rocksdb: CURRENT file: CURRENT -177> 2018-07-21 21:20:44.839012 7f3d23173d80 4 rocksdb: IDENTITY file: IDENTITY -176> 2018-07-21 21:20:44.839018 7f3d23173d80 4 rocksdb: MANIFEST file: MANIFEST-000157 size: 350 Bytes -175> 2018-07-21 21:20:44.839021 7f3d23173d80 4 rocksdb: SST files in db dir, Total Num: 6, files: 000004.sst 000010.sst 000013.sst 000019.sst 000021.sst 000024.sst -174> 2018-07-21 21:20:44.839025 7f3d23173d80 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: -173> 2018-07-21 21:20:44.839030 7f3d23173d80 4 rocksdb: Write Ahead Log file in db.wal: 000158.log size: 0 ; -172> 2018-07-21 21:20:44.839033 7f3d23173d80 4 rocksdb: Options.error_if_exists: 0 -171> 2018-07-21 21:20:44.839041 7f3d23173d80 4 rocksdb: Options.create_if_missing: 0 -170> 2018-07-21 21:20:44.839043 7f3d23173d80 4 rocksdb: Options.paranoid_checks: 1 -169> 2018-07-21 21:20:44.839044 7f3d23173d80 4 rocksdb: Options.env: 0x55ad4e38d0a0 -168> 2018-07-21 21:20:44.839045 7f3d23173d80 4 rocksdb: Options.info_log: 0x55ad4e38d520 -167> 2018-07-21 21:20:44.839046 7f3d23173d80 4 rocksdb: Options.max_open_files: -1 -166> 2018-07-21 21:20:44.839047 7f3d23173d80 4 rocksdb: Options.max_file_opening_threads: 16 -165> 2018-07-21 21:20:44.839048 7f3d23173d80 4 rocksdb: Options.use_fsync: 0 -164> 2018-07-21 21:20:44.839049 7f3d23173d80 4 rocksdb: Options.max_log_file_size: 0 -163> 2018-07-21 21:20:44.839050 7f3d23173d80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -162> 2018-07-21 21:20:44.839051 7f3d23173d80 4 rocksdb: Options.log_file_time_to_roll: 0 -161> 2018-07-21 21:20:44.839052 7f3d23173d80 4 rocksdb: Options.keep_log_file_num: 1000 -160> 2018-07-21 21:20:44.839053 7f3d23173d80 4 rocksdb: Options.recycle_log_file_num: 4 -159> 2018-07-21 21:20:44.839054 7f3d23173d80 4 rocksdb: Options.allow_fallocate: 1 -158> 2018-07-21 21:20:44.839054 7f3d23173d80 4 rocksdb: Options.allow_mmap_reads: 0 -157> 2018-07-21 21:20:44.839055 7f3d23173d80 4 rocksdb: Options.allow_mmap_writes: 0 -156> 2018-07-21 21:20:44.839056 7f3d23173d80 4 rocksdb: Options.use_direct_reads: 0 -155> 2018-07-21 21:20:44.839057 7f3d23173d80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -154> 2018-07-21 21:20:44.839058 7f3d23173d80 4 rocksdb: Options.create_missing_column_families: 0 -153> 2018-07-21 21:20:44.839058 7f3d23173d80 4 rocksdb: Options.db_log_dir: -152> 2018-07-21 21:20:44.839059 7f3d23173d80 4 rocksdb: Options.wal_dir: db.wal -151> 2018-07-21 21:20:44.839060 7f3d23173d80 4 rocksdb: Options.table_cache_numshardbits: 6 -150> 2018-07-21 21:20:44.839061 7f3d23173d80 4 rocksdb: Options.max_subcompactions: 1 -149> 2018-07-21 21:20:44.839062 7f3d23173d80 4 rocksdb: Options.max_background_flushes: 1 -148> 2018-07-21 21:20:44.839062 7f3d23173d80 4 rocksdb: Options.WAL_ttl_seconds: 0 -147> 2018-07-21 21:20:44.839063 7f3d23173d80 4 rocksdb: Options.WAL_size_limit_MB: 0 -146> 2018-07-21 21:20:44.839064 7f3d23173d80 4 rocksdb: Options.manifest_preallocation_size: 4194304 -145> 2018-07-21 21:20:44.839065 7f3d23173d80 4 rocksdb: Options.is_fd_close_on_exec: 1 -144> 2018-07-21 21:20:44.839066 7f3d23173d80 4 rocksdb: Options.advise_random_on_open: 1 -143> 2018-07-21 21:20:44.839066 7f3d23173d80 4 rocksdb: Options.db_write_buffer_size: 0 -142> 2018-07-21 21:20:44.839067 7f3d23173d80 4 rocksdb: Options.access_hint_on_compaction_start: 1 -141> 2018-07-21 21:20:44.839068 7f3d23173d80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -140> 2018-07-21 21:20:44.839069 7f3d23173d80 4 rocksdb: Options.compaction_readahead_size: 2097152 -139> 2018-07-21 21:20:44.839069 7f3d23173d80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -138> 2018-07-21 21:20:44.839070 7f3d23173d80 4 rocksdb: Options.writable_file_max_buffer_size: 0 -137> 2018-07-21 21:20:44.839071 7f3d23173d80 4 rocksdb: Options.use_adaptive_mutex: 0 -136> 2018-07-21 21:20:44.839072 7f3d23173d80 4 rocksdb: Options.rate_limiter: (nil) -135> 2018-07-21 21:20:44.839077 7f3d23173d80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -134> 2018-07-21 21:20:44.839079 7f3d23173d80 4 rocksdb: Options.bytes_per_sync: 0 -133> 2018-07-21 21:20:44.839079 7f3d23173d80 4 rocksdb: Options.wal_bytes_per_sync: 0 -132> 2018-07-21 21:20:44.839080 7f3d23173d80 4 rocksdb: Options.wal_recovery_mode: 0 -131> 2018-07-21 21:20:44.839082 7f3d23173d80 4 rocksdb: Options.enable_thread_tracking: 0 -130> 2018-07-21 21:20:44.839082 7f3d23173d80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -129> 2018-07-21 21:20:44.839083 7f3d23173d80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -128> 2018-07-21 21:20:44.839084 7f3d23173d80 4 rocksdb: Options.write_thread_max_yield_usec: 100 -127> 2018-07-21 21:20:44.839085 7f3d23173d80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -126> 2018-07-21 21:20:44.839085 7f3d23173d80 4 rocksdb: Options.row_cache: None -125> 2018-07-21 21:20:44.839086 7f3d23173d80 4 rocksdb: Options.wal_filter: None -124> 2018-07-21 21:20:44.839087 7f3d23173d80 4 rocksdb: Options.avoid_flush_during_recovery: 0 -123> 2018-07-21 21:20:44.839088 7f3d23173d80 4 rocksdb: Options.base_background_compactions: 1 -122> 2018-07-21 21:20:44.839089 7f3d23173d80 4 rocksdb: Options.max_background_compactions: 1 -121> 2018-07-21 21:20:44.839089 7f3d23173d80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -120> 2018-07-21 21:20:44.839090 7f3d23173d80 4 rocksdb: Options.delayed_write_rate : 16777216 -119> 2018-07-21 21:20:44.839091 7f3d23173d80 4 rocksdb: Options.max_total_wal_size: 0 -118> 2018-07-21 21:20:44.839092 7f3d23173d80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -117> 2018-07-21 21:20:44.839093 7f3d23173d80 4 rocksdb: Options.stats_dump_period_sec: 600 -116> 2018-07-21 21:20:44.839094 7f3d23173d80 4 rocksdb: Compression algorithms supported: -115> 2018-07-21 21:20:44.839095 7f3d23173d80 4 rocksdb: Snappy supported: 1 -114> 2018-07-21 21:20:44.839096 7f3d23173d80 4 rocksdb: Zlib supported: 0 -113> 2018-07-21 21:20:44.839096 7f3d23173d80 4 rocksdb: Bzip supported: 0 -112> 2018-07-21 21:20:44.839097 7f3d23173d80 4 rocksdb: LZ4 supported: 0 -111> 2018-07-21 21:20:44.839098 7f3d23173d80 4 rocksdb: ZSTD supported: 0 -110> 2018-07-21 21:20:44.839099 7f3d23173d80 4 rocksdb: Fast CRC32 supported: 1 -109> 2018-07-21 21:20:44.839340 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000157 -108> 2018-07-21 21:20:44.839413 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: -107> 2018-07-21 21:20:44.839423 7f3d23173d80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -106> 2018-07-21 21:20:44.839431 7f3d23173d80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -105> 2018-07-21 21:20:44.839433 7f3d23173d80 4 rocksdb: Options.compaction_filter: None -104> 2018-07-21 21:20:44.839435 7f3d23173d80 4 rocksdb: Options.compaction_filter_factory: None -103> 2018-07-21 21:20:44.839436 7f3d23173d80 4 rocksdb: Options.memtable_factory: SkipListFactory -102> 2018-07-21 21:20:44.839439 7f3d23173d80 4 rocksdb: Options.table_factory: BlockBasedTable -101> 2018-07-21 21:20:44.839460 7f3d23173d80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ad4e01e228) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55ad4e4900f8 block_cache_name: LRUCache block_cache_options: capacity : 207618048 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 -100> 2018-07-21 21:20:44.839465 7f3d23173d80 4 rocksdb: Options.write_buffer_size: 268435456 -99> 2018-07-21 21:20:44.839471 7f3d23173d80 4 rocksdb: Options.max_write_buffer_number: 4 -98> 2018-07-21 21:20:44.839473 7f3d23173d80 4 rocksdb: Options.compression: NoCompression -97> 2018-07-21 21:20:44.839476 7f3d23173d80 4 rocksdb: Options.bottommost_compression: Disabled -96> 2018-07-21 21:20:44.839478 7f3d23173d80 4 rocksdb: Options.prefix_extractor: nullptr -95> 2018-07-21 21:20:44.839479 7f3d23173d80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -94> 2018-07-21 21:20:44.839481 7f3d23173d80 4 rocksdb: Options.num_levels: 7 -93> 2018-07-21 21:20:44.839482 7f3d23173d80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -92> 2018-07-21 21:20:44.839483 7f3d23173d80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -91> 2018-07-21 21:20:44.839485 7f3d23173d80 4 rocksdb: Options.compression_opts.window_bits: -14 -90> 2018-07-21 21:20:44.839486 7f3d23173d80 4 rocksdb: Options.compression_opts.level: -1 -89> 2018-07-21 21:20:44.839488 7f3d23173d80 4 rocksdb: Options.compression_opts.strategy: 0 -88> 2018-07-21 21:20:44.839489 7f3d23173d80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -87> 2018-07-21 21:20:44.839491 7f3d23173d80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -86> 2018-07-21 21:20:44.839492 7f3d23173d80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -85> 2018-07-21 21:20:44.839493 7f3d23173d80 4 rocksdb: Options.level0_stop_writes_trigger: 36 -84> 2018-07-21 21:20:44.839494 7f3d23173d80 4 rocksdb: Options.target_file_size_base: 67108864 -83> 2018-07-21 21:20:44.839496 7f3d23173d80 4 rocksdb: Options.target_file_size_multiplier: 1 -82> 2018-07-21 21:20:44.839497 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -81> 2018-07-21 21:20:44.839498 7f3d23173d80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -80> 2018-07-21 21:20:44.839500 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -79> 2018-07-21 21:20:44.839503 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -78> 2018-07-21 21:20:44.839505 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -77> 2018-07-21 21:20:44.839507 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -76> 2018-07-21 21:20:44.839508 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -75> 2018-07-21 21:20:44.839509 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -74> 2018-07-21 21:20:44.839511 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -73> 2018-07-21 21:20:44.839512 7f3d23173d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -72> 2018-07-21 21:20:44.839513 7f3d23173d80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -71> 2018-07-21 21:20:44.839515 7f3d23173d80 4 rocksdb: Options.max_compaction_bytes: 1677721600 -70> 2018-07-21 21:20:44.839517 7f3d23173d80 4 rocksdb: Options.arena_block_size: 33554432 -69> 2018-07-21 21:20:44.839518 7f3d23173d80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -68> 2018-07-21 21:20:44.839519 7f3d23173d80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -67> 2018-07-21 21:20:44.839521 7f3d23173d80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -66> 2018-07-21 21:20:44.839522 7f3d23173d80 4 rocksdb: Options.disable_auto_compactions: 0 -65> 2018-07-21 21:20:44.839524 7f3d23173d80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -64> 2018-07-21 21:20:44.839526 7f3d23173d80 4 rocksdb: Options.compaction_pri: kByCompensatedSize -63> 2018-07-21 21:20:44.839527 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -62> 2018-07-21 21:20:44.839529 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -61> 2018-07-21 21:20:44.839530 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -60> 2018-07-21 21:20:44.839532 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -59> 2018-07-21 21:20:44.839533 7f3d23173d80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -58> 2018-07-21 21:20:44.839535 7f3d23173d80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -57> 2018-07-21 21:20:44.839536 7f3d23173d80 4 rocksdb: Options.table_properties_collectors: -56> 2018-07-21 21:20:44.839537 7f3d23173d80 4 rocksdb: Options.inplace_update_support: 0 -55> 2018-07-21 21:20:44.839539 7f3d23173d80 4 rocksdb: Options.inplace_update_num_locks: 10000 -54> 2018-07-21 21:20:44.839540 7f3d23173d80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -53> 2018-07-21 21:20:44.839542 7f3d23173d80 4 rocksdb: Options.memtable_huge_page_size: 0 -52> 2018-07-21 21:20:44.839544 7f3d23173d80 4 rocksdb: Options.bloom_locality: 0 -51> 2018-07-21 21:20:44.839545 7f3d23173d80 4 rocksdb: Options.max_successive_merges: 0 -50> 2018-07-21 21:20:44.839546 7f3d23173d80 4 rocksdb: Options.optimize_filters_for_hits: 0 -49> 2018-07-21 21:20:44.839547 7f3d23173d80 4 rocksdb: Options.paranoid_file_checks: 0 -48> 2018-07-21 21:20:44.839548 7f3d23173d80 4 rocksdb: Options.force_consistency_checks: 0 -47> 2018-07-21 21:20:44.839549 7f3d23173d80 4 rocksdb: Options.report_bg_io_stats: 0 -46> 2018-07-21 21:20:44.841488 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000157 succeeded,manifest_file_number is 157, next_file_number is 159, last_sequence is 253029, log_number is 0,prev_log_number is 0,max_column_family is 0 -45> 2018-07-21 21:20:44.841501 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 156 -44> 2018-07-21 21:20:44.841556 7f3d23173d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844841547, "job": 1, "event": "recovery_started", "log_files": [158]} -43> 2018-07-21 21:20:44.841563 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:482] Recovering log #158 mode 0 -42> 2018-07-21 21:20:44.841577 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2395] Creating manifest 160 -41> 2018-07-21 21:20:44.842391 7f3d23173d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844842388, "job": 1, "event": "recovery_finished"} -40> 2018-07-21 21:20:44.842447 7f3d23173d80 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db.wal/000158.log type=0 #158 -- OK -39> 2018-07-21 21:20:44.842459 7f3d23173d80 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//MANIFEST-000157 type=3 #157 -- OK -38> 2018-07-21 21:20:44.845424 7f3d23173d80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55ad4e2c2b00 -37> 2018-07-21 21:20:44.845488 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -36> 2018-07-21 21:20:44.845760 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- -35> 2018-07-21 21:20:44.845790 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:448] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 6/0 4.77 MB 1.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 6/0 4.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 6/0 4.77 MB 1.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 6/0 4.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** -34> 2018-07-21 21:20:44.845926 7f3d0e782700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 6@0 files to L1, score 1.50 -33> 2018-07-21 21:20:44.845954 7f3d0e782700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 0, inputs: [24(1406B) 21(14KB) 19(4561KB) 13(308KB) 10(1391B) 4(1456B)] -32> 2018-07-21 21:20:44.846000 7f3d0e782700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1532200844845965, "job": 3, "event": "compaction_started", "files_L0": [24, 21, 19, 13, 10, 4], "score": 1.5, "input_data_size": 5006316} -31> 2018-07-21 21:20:44.846921 7f3d23173d80 1 freelist init -30> 2018-07-21 21:20:44.847517 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_alloc opening allocation metadata -29> 2018-07-21 21:20:44.847652 7f3d23173d80 1 bluestore(/var/lib/ceph/osd/ceph-24) _open_alloc loaded 1787 G in 1 extents -28> 2018-07-21 21:20:44.853191 7f3d23173d80 2 osd.24 0 journal looks like ssd -27> 2018-07-21 21:20:44.853206 7f3d23173d80 2 osd.24 0 boot -26> 2018-07-21 21:20:44.854250 7f3d23173d80 0 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -25> 2018-07-21 21:20:44.854289 7f3d23173d80 0 _get_class not permitted to load lua -24> 2018-07-21 21:20:44.854303 7f3d23173d80 0 _get_class not permitted to load kvs -23> 2018-07-21 21:20:44.854780 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class! -22> 2018-07-21 21:20:44.855020 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/statelog/cls_statelog.cc:279: Loaded log class! -21> 2018-07-21 21:20:44.855429 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/user/cls_user.cc:410: Loaded user class! -20> 2018-07-21 21:20:44.859325 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/rgw/cls_rgw.cc:3831: Loaded rgw class! -19> 2018-07-21 21:20:44.859963 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/log/cls_log.cc:299: Loaded log class! -18> 2018-07-21 21:20:44.859996 7f3d23173d80 0 _get_class not permitted to load sdk -17> 2018-07-21 21:20:44.860221 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -16> 2018-07-21 21:20:44.860460 7f3d23173d80 0 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello -15> 2018-07-21 21:20:44.864106 7f3d0e782700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1591] Compaction error: Corruption: block checksum mismatch -14> 2018-07-21 21:20:44.864130 7f3d0e782700 4 rocksdb: (Original Log Time 2018/07/21-21:20:44.864026) [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[6 0 0 0 0 0 0] max score 0.00, MB/sec: 278.4 rd, 20.3 wr, level 1, files in(6, 0) out(1) MB in(4.8, 0.0) out(0.3), read-write-amplify(1.1) write-amplify(0.1) Corruption: block checksum mismatch, records in: 4782 -13> 2018-07-21 21:20:44.864140 7f3d0e782700 4 rocksdb: (Original Log Time 2018/07/21-21:20:44.864089) EVENT_LOG_v1 {"time_micros": 1532200844864063, "job": 3, "event": "compaction_finished", "compaction_time_micros": 17983, "output_level": 1, "num_output_files": 1, "total_output_size": 365633, "num_input_records": 3539, "num_output_records": 1817, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 0, 0, 0, 0, 0, 0]} -12> 2018-07-21 21:20:44.864146 7f3d0e782700 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1275] Waiting after background compaction error: Corruption: block checksum mismatch, Accumulated background error counts: 1 -11> 2018-07-21 21:20:44.864464 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/refcount/cls_refcount.cc:222: Loaded refcount class! -10> 2018-07-21 21:20:44.864682 7f3d23173d80 1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/cls/version/cls_version.cc:214: Loaded version class! -9> 2018-07-21 21:20:44.865208 7f3d23173d80 0 osd.24 763 crush map has features 288514051259236352, adjusting msgr requires for clients -8> 2018-07-21 21:20:44.865220 7f3d23173d80 0 osd.24 763 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons -7> 2018-07-21 21:20:44.865226 7f3d23173d80 0 osd.24 763 crush map has features 1009089991638532096, adjusting msgr requires for osds -6> 2018-07-21 21:20:44.873029 7f3d23173d80 0 osd.24 763 load_pgs -5> 2018-07-21 21:20:44.873582 7f3d23173d80 5 osd.24 pg_epoch: 763 pg[9.e(unlocked)] enter Initial -4> 2018-07-21 21:20:44.907499 7f3d23173d80 5 osd.24 pg_epoch: 763 pg[9.e( v 763'1570 (0'0,763'1570] local-lis/les=758/759 n=2 ec=729/729 lis/c 758/758 les/c/f 759/759/0 758/758/755) [11,25,24] r=2 lpr=0 crt=763'1570 lcod 0'0 unknown NOTIFY] exit Initial 0.033931 0 0.000000 -3> 2018-07-21 21:20:44.907562 7f3d23173d80 5 osd.24 pg_epoch: 763 pg[9.e( v 763'1570 (0'0,763'1570] local-lis/les=758/759 n=2 ec=729/729 lis/c 758/758 les/c/f 759/759/0 758/758/755) [11,25,24] r=2 lpr=0 crt=763'1570 lcod 0'0 unknown NOTIFY] enter Reset -2> 2018-07-21 21:20:44.907595 7f3d23173d80 5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 -1> 2018-07-21 21:20:44.907849 7f3d0ff85700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = O key = 0x7f80000000000000097000000021213dfffffffffffffffeffffffffffffffff'o' Value size = 29) Put( Prefix = S key = 'nid_max' Value size = 8) Put( Prefix = S key = 'blobid_max' Value size = 8) 0> 2018-07-21 21:20:44.911270 7f3d0ff85700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f3d0ff85700 time 2018-07-21 21:20:44.907898 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueStore.cc: 8526: FAILED assert(r == 0) ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55ad44902850] 2: (BlueStore::_kv_sync_thread()+0x3472) [0x55ad447b1042] 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ad447f792d] 4: (()+0x7e25) [0x7f3d20845e25] 5: (clone()+0x6d) [0x7f3d1f936bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.24.log --- end dump of recent events --- 2018-07-21 21:20:44.917216 7f3d0ff85700 -1 *** Caught signal (Aborted) ** in thread 7f3d0ff85700 thread_name:bstore_kv_sync ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0xa48ec1) [0x55ad448c3ec1] 2: (()+0xf6d0) [0x7f3d2084d6d0] 3: (gsignal()+0x37) [0x7f3d1f86e277] 4: (abort()+0x148) [0x7f3d1f86f968] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55ad449029c4] 6: (BlueStore::_kv_sync_thread()+0x3472) [0x55ad447b1042] 7: (BlueStore::KVSyncThread::entry()+0xd) [0x55ad447f792d] 8: (()+0x7e25) [0x7f3d20845e25] 9: (clone()+0x6d) [0x7f3d1f936bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-07-21 21:20:44.917216 7f3d0ff85700 -1 *** Caught signal (Aborted) ** in thread 7f3d0ff85700 thread_name:bstore_kv_sync ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0xa48ec1) [0x55ad448c3ec1] 2: (()+0xf6d0) [0x7f3d2084d6d0] 3: (gsignal()+0x37) [0x7f3d1f86e277] 4: (abort()+0x148) [0x7f3d1f86f968] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55ad449029c4] 6: (BlueStore::_kv_sync_thread()+0x3472) [0x55ad447b1042] 7: (BlueStore::KVSyncThread::entry()+0xd) [0x55ad447f792d] 8: (()+0x7e25) [0x7f3d20845e25] 9: (clone()+0x6d) [0x7f3d1f936bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.24.log --- end dump of recent events ---