Project

General

Profile

Bug #16278 » ceph-osd-3.log

Mikaël Cluseau, 06/14/2016 04:56 AM

 
+ image=ceph/daemon:jewel
+ '[' 3 = 3 ']'
+ image=ceph/daemon:tag-build-master-jewel-ubuntu-16.04
+ LABEL=ceph-osd-3
+ mp=/var/lib/ceph/osd/ceph-3
+ mkdir -p /var/lib/ceph/osd/ceph-3
+ umount /var/lib/ceph/osd/ceph-3
+ fsck -p LABEL=ceph-osd-3
fsck from util-linux 2.26.2
/sbin/fsck.xfs: XFS file system.
+ mount -o noatime,nodiratime,relatime LABEL=ceph-osd-3 /var/lib/ceph/osd/ceph-3
+ docker kill ceph-osd-3
Error response from daemon: Cannot kill container ceph-osd-3: nosuchcontainer: no such id: ceph-osd-3
Error: failed to kill containers: [ceph-osd-3]
+ true
+ docker rm ceph-osd-3
Error response from daemon: no such id: ceph-osd-3
Error: failed to remove containers: [ceph-osd-3]
+ true
++ hostname -s
+ exec /usr/bin/docker run --name=ceph-osd-3 --rm -v /var/lib/ceph/osd/ceph-3:/var/lib/ceph/osd/ceph-3 -v /etc/ceph:/etc/ceph -v /dev/:/dev/ -e HOSTNAME=ceph-2 --net=host --privileged ceph/daemon:tag-build-master-jewel-ubuntu-16.04 osd
static: does not generate config
Bootstrapped OSD(s) found; using OSD directory
*** Running /etc/rc.local...
*** Booting runit daemon...
*** Runit started as PID 22
store-daemon: starting daemon on ceph-2...
2016-06-14 04:51:53.712282 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-06-14 04:51:53.712356 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-06-14 04:51:53.712374 7faa48c7a8c0 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 24
starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
2016-06-14 04:51:53.724872 7faa48c7a8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

2016-06-14 04:51:53.726844 7faa48c7a8c0 0 pidfile_write: ignore empty --pid-file
2016-06-14 04:51:53.741965 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-06-14 04:51:53.850122 7faa48c7a8c0 1 bluestore(/var/lib/ceph/osd/ceph-3) mount path /var/lib/ceph/osd/ceph-3
2016-06-14 04:51:53.862577 7faa48c7a8c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2016-06-14 04:51:53.868966 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2016-06-14 04:51:53.908830 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
2016-06-14 04:51:53.920381 7faa48c7a8c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2016-06-14 04:51:53.921877 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2016-06-14 04:51:53.928452 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
2016-06-14 04:51:53.930279 7faa48c7a8c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 931 GB
2016-06-14 04:51:53.936332 7faa48c7a8c0 1 bluefs mount
2016-06-14 04:51:54.229914 7faa48c7a8c0 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

2016-06-14 04:51:54.238598 7faa48c7a8c0 0 set rocksdb option compression = kNoCompression
2016-06-14 04:51:54.238847 7faa48c7a8c0 0 set rocksdb option max_write_buffer_number = 16
2016-06-14 04:51:54.239096 7faa48c7a8c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-06-14 04:51:54.250300 7faa48c7a8c0 0 set rocksdb option recycle_log_file_num = 16
2016-06-14 04:51:54.250608 7faa48c7a8c0 0 set rocksdb option compression = kNoCompression
2016-06-14 04:51:54.250864 7faa48c7a8c0 0 set rocksdb option max_write_buffer_number = 16
2016-06-14 04:51:54.256286 7faa48c7a8c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
2016-06-14 04:51:54.256506 7faa48c7a8c0 0 set rocksdb option recycle_log_file_num = 16
2016-06-14 04:51:54.256852 7faa48c7a8c0 4 rocksdb: RocksDB version: 4.3.0

2016-06-14 04:51:54.257101 7faa48c7a8c0 4 rocksdb: Git sha rocksdb_build_git_sha:
2016-06-14 04:51:54.257324 7faa48c7a8c0 4 rocksdb: Compile date May 13 2016
2016-06-14 04:51:54.257549 7faa48c7a8c0 4 rocksdb: DB SUMMARY

2016-06-14 04:51:54.286950 7faa48c7a8c0 4 rocksdb: CURRENT file: CURRENT

2016-06-14 04:51:54.292424 7faa48c7a8c0 4 rocksdb: IDENTITY file: IDENTITY

2016-06-14 04:51:54.292665 7faa48c7a8c0 4 rocksdb: MANIFEST file: MANIFEST-028105 size: 9090 Bytes

2016-06-14 04:51:54.293822 7faa48c7a8c0 4 rocksdb: SST files in db dir, Total Num: 62, files: 000599.sst 000600.sst 000601.sst 000771.sst 000772.sst 000773.sst 000774.sst 000852.sst 000853.sst

2016-06-14 04:51:54.305195 7faa48c7a8c0 4 rocksdb: Write Ahead Log file in db: 028106.log size: 0 ;

2016-06-14 04:51:54.311594 7faa48c7a8c0 4 rocksdb: Options.error_if_exists: 0
2016-06-14 04:51:54.311819 7faa48c7a8c0 4 rocksdb: Options.create_if_missing: 0
2016-06-14 04:51:54.312074 7faa48c7a8c0 4 rocksdb: Options.paranoid_checks: 1
2016-06-14 04:51:54.312320 7faa48c7a8c0 4 rocksdb: Options.env: 0x55f1c054c0c0
2016-06-14 04:51:54.318269 7faa48c7a8c0 4 rocksdb: Options.info_log: 0x55f1c054c0a0
2016-06-14 04:51:54.318474 7faa48c7a8c0 4 rocksdb: Options.max_open_files: 5000
2016-06-14 04:51:54.318703 7faa48c7a8c0 4 rocksdb: Options.max_file_opening_threads: 1
2016-06-14 04:51:54.318934 7faa48c7a8c0 4 rocksdb: Options.max_total_wal_size: 0
2016-06-14 04:51:54.319147 7faa48c7a8c0 4 rocksdb: Options.disableDataSync: 0
2016-06-14 04:51:54.319382 7faa48c7a8c0 4 rocksdb: Options.use_fsync: 0
2016-06-14 04:51:54.319582 7faa48c7a8c0 4 rocksdb: Options.max_log_file_size: 0
2016-06-14 04:51:54.325289 7faa48c7a8c0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2016-06-14 04:51:54.325529 7faa48c7a8c0 4 rocksdb: Options.log_file_time_to_roll: 0
2016-06-14 04:51:54.325728 7faa48c7a8c0 4 rocksdb: Options.keep_log_file_num: 1000
2016-06-14 04:51:54.325955 7faa48c7a8c0 4 rocksdb: Options.recycle_log_file_num: 16
2016-06-14 04:51:54.326188 7faa48c7a8c0 4 rocksdb: Options.allow_os_buffer: 1
2016-06-14 04:51:54.332241 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_reads: 0
2016-06-14 04:51:54.338238 7faa48c7a8c0 4 rocksdb: Options.allow_fallocate: 1
2016-06-14 04:51:54.338469 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_writes: 0
2016-06-14 04:51:54.338695 7faa48c7a8c0 4 rocksdb: Options.create_missing_column_families: 0
2016-06-14 04:51:54.338902 7faa48c7a8c0 4 rocksdb: Options.db_log_dir:
2016-06-14 04:51:54.339145 7faa48c7a8c0 4 rocksdb: Options.wal_dir: db
2016-06-14 04:51:54.345269 7faa48c7a8c0 4 rocksdb: Options.table_cache_numshardbits: 4
2016-06-14 04:51:54.345465 7faa48c7a8c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2016-06-14 04:51:54.345694 7faa48c7a8c0 4 rocksdb: Options.max_background_compactions: 1
2016-06-14 04:51:54.345900 7faa48c7a8c0 4 rocksdb: Options.max_subcompactions: 1
2016-06-14 04:51:54.346161 7faa48c7a8c0 4 rocksdb: Options.max_background_flushes: 1
2016-06-14 04:51:54.346385 7faa48c7a8c0 4 rocksdb: Options.WAL_ttl_seconds: 0
2016-06-14 04:51:54.352294 7faa48c7a8c0 4 rocksdb: Options.WAL_size_limit_MB: 0
2016-06-14 04:51:54.352561 7faa48c7a8c0 4 rocksdb: Options.manifest_preallocation_size: 4194304
2016-06-14 04:51:54.352762 7faa48c7a8c0 4 rocksdb: Options.allow_os_buffer: 1
2016-06-14 04:51:54.353007 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_reads: 0
2016-06-14 04:51:54.353239 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_writes: 0
2016-06-14 04:51:54.353434 7faa48c7a8c0 4 rocksdb: Options.is_fd_close_on_exec: 1
2016-06-14 04:51:54.353674 7faa48c7a8c0 4 rocksdb: Options.stats_dump_period_sec: 600
2016-06-14 04:51:54.365191 7faa48c7a8c0 4 rocksdb: Options.advise_random_on_open: 1
2016-06-14 04:51:54.365421 7faa48c7a8c0 4 rocksdb: Options.db_write_buffer_size: 0d
2016-06-14 04:51:54.365618 7faa48c7a8c0 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
2016-06-14 04:51:54.365856 7faa48c7a8c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2016-06-14 04:51:54.366052 7faa48c7a8c0 4 rocksdb: Options.compaction_readahead_size: 0d
2016-06-14 04:51:54.366299 7faa48c7a8c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
2016-06-14 04:51:54.372240 7faa48c7a8c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
2016-06-14 04:51:54.372504 7faa48c7a8c0 4 rocksdb: Options.use_adaptive_mutex: 0
2016-06-14 04:51:54.372701 7faa48c7a8c0 4 rocksdb: Options.rate_limiter: (nil)
2016-06-14 04:51:54.372971 7faa48c7a8c0 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
2016-06-14 04:51:54.373192 7faa48c7a8c0 4 rocksdb: Options.bytes_per_sync: 0
2016-06-14 04:51:54.373407 7faa48c7a8c0 4 rocksdb: Options.wal_bytes_per_sync: 0
2016-06-14 04:51:54.379290 7faa48c7a8c0 4 rocksdb: Options.wal_recovery_mode: 0
2016-06-14 04:51:54.379537 7faa48c7a8c0 4 rocksdb: Options.enable_thread_tracking: 0
2016-06-14 04:51:54.379744 7faa48c7a8c0 4 rocksdb: Options.row_cache: None
2016-06-14 04:51:54.380011 7faa48c7a8c0 4 rocksdb: Options.wal_filter: None
2016-06-14 04:51:54.380236 7faa48c7a8c0 4 rocksdb: Compression algorithms supported:
2016-06-14 04:51:54.386222 7faa48c7a8c0 4 rocksdb: Snappy supported: 1
2016-06-14 04:51:54.386461 7faa48c7a8c0 4 rocksdb: Zlib supported: 1
2016-06-14 04:51:54.392176 7faa48c7a8c0 4 rocksdb: Bzip supported: 0
2016-06-14 04:51:54.392403 7faa48c7a8c0 4 rocksdb: LZ4 supported: 0
2016-06-14 04:51:54.392620 7faa48c7a8c0 4 rocksdb: Fast CRC32 supported: 0
2016-06-14 04:51:54.394593 7faa48c7a8c0 4 rocksdb: Recovering from manifest file: MANIFEST-028105

2016-06-14 04:51:54.394896 7faa48c7a8c0 4 rocksdb: --------------- Options for column family [default]:

2016-06-14 04:51:54.395152 7faa48c7a8c0 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2016-06-14 04:51:54.395385 7faa48c7a8c0 4 rocksdb: Options.merge_operator: None
2016-06-14 04:51:54.395596 7faa48c7a8c0 4 rocksdb: Options.compaction_filter: None
2016-06-14 04:51:54.407254 7faa48c7a8c0 4 rocksdb: Options.compaction_filter_factory: None
2016-06-14 04:51:54.418722 7faa48c7a8c0 4 rocksdb: Options.memtable_factory: SkipListFactory
2016-06-14 04:51:54.418975 7faa48c7a8c0 4 rocksdb: Options.table_factory: BlockBasedTable
2016-06-14 04:51:54.419241 7faa48c7a8c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f1bfdcc108)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x55f1bfdf5840
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0

2016-06-14 04:51:54.431426 7faa48c7a8c0 4 rocksdb: Options.write_buffer_size: 4194304
2016-06-14 04:51:54.439305 7faa48c7a8c0 4 rocksdb: Options.max_write_buffer_number: 16
2016-06-14 04:51:54.439563 7faa48c7a8c0 4 rocksdb: Options.compression: NoCompression
2016-06-14 04:51:54.439779 7faa48c7a8c0 4 rocksdb: Options.prefix_extractor: nullptr
2016-06-14 04:51:54.440019 7faa48c7a8c0 4 rocksdb: Options.num_levels: 7
2016-06-14 04:51:54.440237 7faa48c7a8c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
2016-06-14 04:51:54.440456 7faa48c7a8c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2016-06-14 04:51:54.440667 7faa48c7a8c0 4 rocksdb: Options.compression_opts.window_bits: -14
2016-06-14 04:51:54.440897 7faa48c7a8c0 4 rocksdb: Options.compression_opts.level: -1
2016-06-14 04:51:54.441140 7faa48c7a8c0 4 rocksdb: Options.compression_opts.strategy: 0
2016-06-14 04:51:54.441339 7faa48c7a8c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2016-06-14 04:51:54.441561 7faa48c7a8c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2016-06-14 04:51:54.441772 7faa48c7a8c0 4 rocksdb: Options.level0_stop_writes_trigger: 24
2016-06-14 04:51:54.441994 7faa48c7a8c0 4 rocksdb: Options.target_file_size_base: 2097152
2016-06-14 04:51:54.442220 7faa48c7a8c0 4 rocksdb: Options.target_file_size_multiplier: 1
2016-06-14 04:51:54.442432 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_base: 10485760
2016-06-14 04:51:54.442654 7faa48c7a8c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2016-06-14 04:51:54.442902 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
2016-06-14 04:51:54.443142 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2016-06-14 04:51:54.443341 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2016-06-14 04:51:54.443563 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2016-06-14 04:51:54.474333 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2016-06-14 04:51:54.478289 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2016-06-14 04:51:54.478539 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2016-06-14 04:51:54.478757 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2016-06-14 04:51:54.478983 7faa48c7a8c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2016-06-14 04:51:54.479217 7faa48c7a8c0 4 rocksdb: Options.expanded_compaction_factor: 25
2016-06-14 04:51:54.479428 7faa48c7a8c0 4 rocksdb: Options.source_compaction_factor: 1
2016-06-14 04:51:54.479663 7faa48c7a8c0 4 rocksdb: Options.max_grandparent_overlap_factor: 10
2016-06-14 04:51:54.479869 7faa48c7a8c0 4 rocksdb: Options.arena_block_size: 524288
2016-06-14 04:51:54.480091 7faa48c7a8c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
2016-06-14 04:51:54.480304 7faa48c7a8c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
2016-06-14 04:51:54.480529 7faa48c7a8c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2016-06-14 04:51:54.480738 7faa48c7a8c0 4 rocksdb: Options.disable_auto_compactions: 0
2016-06-14 04:51:54.480960 7faa48c7a8c0 4 rocksdb: Options.filter_deletes: 0
2016-06-14 04:51:54.481184 7faa48c7a8c0 4 rocksdb: Options.verify_checksums_in_compaction: 1
2016-06-14 04:51:54.481397 7faa48c7a8c0 4 rocksdb: Options.compaction_style: 0
2016-06-14 04:51:54.481624 7faa48c7a8c0 4 rocksdb: Options.compaction_pri: 0
2016-06-14 04:51:54.481851 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2016-06-14 04:51:54.482078 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2016-06-14 04:51:54.482323 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2016-06-14 04:51:54.482543 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2016-06-14 04:51:54.482770 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2016-06-14 04:51:54.482991 7faa48c7a8c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2016-06-14 04:51:54.495245 7faa48c7a8c0 4 rocksdb: Options.table_properties_collectors:
2016-06-14 04:51:54.522550 7faa48c7a8c0 4 rocksdb: Options.inplace_update_support: 0
2016-06-14 04:51:54.522902 7faa48c7a8c0 4 rocksdb: Options.inplace_update_num_locks: 10000
2016-06-14 04:51:54.523138 7faa48c7a8c0 4 rocksdb: Options.min_partial_merge_operands: 2
2016-06-14 04:51:54.523366 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
2016-06-14 04:51:54.523595 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
2016-06-14 04:51:54.523817 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
2016-06-14 04:51:54.524034 7faa48c7a8c0 4 rocksdb: Options.bloom_locality: 0
2016-06-14 04:51:54.524260 7faa48c7a8c0 4 rocksdb: Options.max_successive_merges: 0
2016-06-14 04:51:54.524481 7faa48c7a8c0 4 rocksdb: Options.optimize_fllters_for_hits: 0
2016-06-14 04:51:54.524704 7faa48c7a8c0 4 rocksdb: Options.paranoid_file_checks: 0
2016-06-14 04:51:54.524923 7faa48c7a8c0 4 rocksdb: Options.compaction_measure_io_stats: 0
2016-06-14 04:51:54.532672 7faa48c7a8c0 4 rocksdb: Recovered from manifest file:db/MANIFEST-028105 succeeded,manifest_file_number is 28105, next_file_number is 28107, last_sequence is 3262218, log_number is 0,prev_log_number is 0,max_column_family is 0

2016-06-14 04:51:54.552661 7faa48c7a8c0 4 rocksdb: Column family [default] (ID 0), log number is 28104

2016-06-14 04:51:54.566063 7faa48c7a8c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1465879914566054, "job": 1, "event": "recovery_started", "log_files": [28106]}
2016-06-14 04:51:54.582565 7faa48c7a8c0 4 rocksdb: Recovering log #28106 mode 0 skip-recovery 0
2016-06-14 04:51:54.582831 7faa48c7a8c0 4 rocksdb: Creating manifest 28108

2016-06-14 04:51:54.583887 7faa48c7a8c0 4 rocksdb: Deleting manifest 28105 current manifest 28108

2016-06-14 04:51:54.584208 7faa48c7a8c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1465879914584200, "job": 1, "event": "recovery_finished"}
2016-06-14 04:51:54.681961 7faa48c7a8c0 4 rocksdb: DB pointer 0x55f1c055a000
2016-06-14 04:51:54.709583 7faa48c7a8c0 1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=16
2016-06-14 04:51:54.709870 7faa48c7a8c0 1 freelist init prefix B
2016-06-14 04:51:54.774695 7faa48c7a8c0 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
2016-06-14 04:51:54.980440 7faa48c7a8c0 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7faa48c7a8c0 time 2016-06-14 04:51:55.000137
osd/OSD.h: 885: FAILED assert(ret)
ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f1b486fd60]
2: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
3: (OSD::init()+0x1f91) [0x55f1b419b391]
4: (main()+0x2ea5) [0x55f1b410cdd5]
5: (__libc_start_main()+0xf0) [0x7faa45abf830]
6: (_start()+0x29) [0x55f1b414e469]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-06-14 04:51:55.017680 7faa48c7a8c0 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7faa48c7a8c0 time 2016-06-14 04:51:55.000137
osd/OSD.h: 885: FAILED assert(ret)

ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f1b486fd60]
2: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
3: (OSD::init()+0x1f91) [0x55f1b419b391]
4: (main()+0x2ea5) [0x55f1b410cdd5]
5: (__libc_start_main()+0xf0) [0x7faa45abf830]
6: (_start()+0x29) [0x55f1b414e469]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-215> 2016-06-14 04:51:53.691517 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command perfcounters_dump hook 0x55f1bfdd0030
-214> 2016-06-14 04:51:53.691551 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command 1 hook 0x55f1bfdd0030
-213> 2016-06-14 04:51:53.691555 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command perf dump hook 0x55f1bfdd0030
-212> 2016-06-14 04:51:53.691558 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command perfcounters_schema hook 0x55f1bfdd0030
-211> 2016-06-14 04:51:53.691560 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command 2 hook 0x55f1bfdd0030
-210> 2016-06-14 04:51:53.691562 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command perf schema hook 0x55f1bfdd0030
-209> 2016-06-14 04:51:53.691565 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command perf reset hook 0x55f1bfdd0030
-208> 2016-06-14 04:51:53.691567 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command config show hook 0x55f1bfdd0030
-207> 2016-06-14 04:51:53.691569 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command config set hook 0x55f1bfdd0030
-206> 2016-06-14 04:51:53.691572 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command config get hook 0x55f1bfdd0030
-205> 2016-06-14 04:51:53.691574 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command config diff hook 0x55f1bfdd0030
-204> 2016-06-14 04:51:53.691576 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command log flush hook 0x55f1bfdd0030
-203> 2016-06-14 04:51:53.691578 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command log dump hook 0x55f1bfdd0030
-202> 2016-06-14 04:51:53.691580 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command log reopen hook 0x55f1bfdd0030
-201> 2016-06-14 04:51:53.712282 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-200> 2016-06-14 04:51:53.712356 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-199> 2016-06-14 04:51:53.712374 7faa48c7a8c0 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 24
-198> 2016-06-14 04:51:53.724846 7faa48c7a8c0 5 object store type is bluestore
-197> 2016-06-14 04:51:53.724872 7faa48c7a8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

-196> 2016-06-14 04:51:53.726775 7faa48c7a8c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/24 need_addr=1
-195> 2016-06-14 04:51:53.726803 7faa48c7a8c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/24 need_addr=1
-194> 2016-06-14 04:51:53.726821 7faa48c7a8c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/24 need_addr=1
-193> 2016-06-14 04:51:53.726837 7faa48c7a8c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/24 need_addr=1
-192> 2016-06-14 04:51:53.726844 7faa48c7a8c0 0 pidfile_write: ignore empty --pid-file
-191> 2016-06-14 04:51:53.741965 7faa48c7a8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-190> 2016-06-14 04:51:53.741975 7faa48c7a8c0 5 asok(0x55f1bfe90000) init /var/run/ceph/ceph-osd.3.asok
-189> 2016-06-14 04:51:53.741986 7faa48c7a8c0 5 asok(0x55f1bfe90000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
-188> 2016-06-14 04:51:53.742217 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command 0 hook 0x55f1bfdcc0a8
-187> 2016-06-14 04:51:53.742225 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command version hook 0x55f1bfdcc0a8
-186> 2016-06-14 04:51:53.742229 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command git_version hook 0x55f1bfdcc0a8
-185> 2016-06-14 04:51:53.742233 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command help hook 0x55f1bfdd0110
-184> 2016-06-14 04:51:53.742236 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command get_command_descriptions hook 0x55f1bfdd0120
-183> 2016-06-14 04:51:53.742302 7faa48c7a8c0 10 monclient(hunting): build_initial_monmap
-182> 2016-06-14 04:51:53.745163 7faa4305a700 5 asok(0x55f1bfe90000) entry start
-181> 2016-06-14 04:51:53.835222 7faa48c7a8c0 5 adding auth protocol: cephx
-180> 2016-06-14 04:51:53.835257 7faa48c7a8c0 5 adding auth protocol: cephx
-179> 2016-06-14 04:51:53.835465 7faa48c7a8c0 5 asok(0x55f1bfe90000) register_command objecter_requests hook 0x55f1bfdd01d0
-178> 2016-06-14 04:51:53.835573 7faa48c7a8c0 1 -- 0.0.0.0:6800/24 messenger.start
-177> 2016-06-14 04:51:53.835628 7faa48c7a8c0 1 -- :/0 messenger.start
-176> 2016-06-14 04:51:53.835661 7faa48c7a8c0 1 -- 0.0.0.0:6803/24 messenger.start
-175> 2016-06-14 04:51:53.835695 7faa48c7a8c0 1 -- 0.0.0.0:6802/24 messenger.start
-174> 2016-06-14 04:51:53.835727 7faa48c7a8c0 1 -- 0.0.0.0:6801/24 messenger.start
-173> 2016-06-14 04:51:53.835758 7faa48c7a8c0 1 -- :/0 messenger.start
-172> 2016-06-14 04:51:53.835847 7faa48c7a8c0 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
-171> 2016-06-14 04:51:53.850122 7faa48c7a8c0 1 bluestore(/var/lib/ceph/osd/ceph-3) mount path /var/lib/ceph/osd/ceph-3
-170> 2016-06-14 04:51:53.862577 7faa48c7a8c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
-169> 2016-06-14 04:51:53.868966 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
-168> 2016-06-14 04:51:53.908830 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
-167> 2016-06-14 04:51:53.920381 7faa48c7a8c0 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
-166> 2016-06-14 04:51:53.921877 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
-165> 2016-06-14 04:51:53.928452 7faa48c7a8c0 1 bdev(/var/lib/ceph/osd/ceph-3/block) open size 1000098962944 (931 GB) block_size 4096 (4096 B)
-164> 2016-06-14 04:51:53.930279 7faa48c7a8c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 931 GB
-163> 2016-06-14 04:51:53.936332 7faa48c7a8c0 1 bluefs mount
-162> 2016-06-14 04:51:54.229914 7faa48c7a8c0 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.

-161> 2016-06-14 04:51:54.238598 7faa48c7a8c0 0 set rocksdb option compression = kNoCompression
-160> 2016-06-14 04:51:54.238847 7faa48c7a8c0 0 set rocksdb option max_write_buffer_number = 16
-159> 2016-06-14 04:51:54.239096 7faa48c7a8c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
-158> 2016-06-14 04:51:54.250300 7faa48c7a8c0 0 set rocksdb option recycle_log_file_num = 16
-157> 2016-06-14 04:51:54.250608 7faa48c7a8c0 0 set rocksdb option compression = kNoCompression
-156> 2016-06-14 04:51:54.250864 7faa48c7a8c0 0 set rocksdb option max_write_buffer_number = 16
-155> 2016-06-14 04:51:54.256286 7faa48c7a8c0 0 set rocksdb option min_write_buffer_number_to_merge = 3
-154> 2016-06-14 04:51:54.256506 7faa48c7a8c0 0 set rocksdb option recycle_log_file_num = 16
-153> 2016-06-14 04:51:54.256852 7faa48c7a8c0 4 rocksdb: RocksDB version: 4.3.0

-152> 2016-06-14 04:51:54.257101 7faa48c7a8c0 4 rocksdb: Git sha rocksdb_build_git_sha:
-151> 2016-06-14 04:51:54.257324 7faa48c7a8c0 4 rocksdb: Compile date May 13 2016
-150> 2016-06-14 04:51:54.257549 7faa48c7a8c0 4 rocksdb: DB SUMMARY

-149> 2016-06-14 04:51:54.286950 7faa48c7a8c0 4 rocksdb: CURRENT file: CURRENT

-148> 2016-06-14 04:51:54.292424 7faa48c7a8c0 4 rocksdb: IDENTITY file: IDENTITY

-147> 2016-06-14 04:51:54.292665 7faa48c7a8c0 4 rocksdb: MANIFEST file: MANIFEST-028105 size: 9090 Bytes

-146> 2016-06-14 04:51:54.293822 7faa48c7a8c0 4 rocksdb: SST files in db dir, Total Num: 62, files: 000599.sst 000600.sst 000601.sst 000771.sst 000772.sst 000773.sst 000774.sst 000852.sst 000853.sst

-145> 2016-06-14 04:51:54.305195 7faa48c7a8c0 4 rocksdb: Write Ahead Log file in db: 028106.log size: 0 ;

-144> 2016-06-14 04:51:54.311594 7faa48c7a8c0 4 rocksdb: Options.error_if_exists: 0
-143> 2016-06-14 04:51:54.311819 7faa48c7a8c0 4 rocksdb: Options.create_if_missing: 0
-142> 2016-06-14 04:51:54.312074 7faa48c7a8c0 4 rocksdb: Options.paranoid_checks: 1
-141> 2016-06-14 04:51:54.312320 7faa48c7a8c0 4 rocksdb: Options.env: 0x55f1c054c0c0
-140> 2016-06-14 04:51:54.318269 7faa48c7a8c0 4 rocksdb: Options.info_log: 0x55f1c054c0a0
-139> 2016-06-14 04:51:54.318474 7faa48c7a8c0 4 rocksdb: Options.max_open_files: 5000
-138> 2016-06-14 04:51:54.318703 7faa48c7a8c0 4 rocksdb: Options.max_file_opening_threads: 1
-137> 2016-06-14 04:51:54.318934 7faa48c7a8c0 4 rocksdb: Options.max_total_wal_size: 0
-136> 2016-06-14 04:51:54.319147 7faa48c7a8c0 4 rocksdb: Options.disableDataSync: 0
-135> 2016-06-14 04:51:54.319382 7faa48c7a8c0 4 rocksdb: Options.use_fsync: 0
-134> 2016-06-14 04:51:54.319582 7faa48c7a8c0 4 rocksdb: Options.max_log_file_size: 0
-133> 2016-06-14 04:51:54.325289 7faa48c7a8c0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-132> 2016-06-14 04:51:54.325529 7faa48c7a8c0 4 rocksdb: Options.log_file_time_to_roll: 0
-131> 2016-06-14 04:51:54.325728 7faa48c7a8c0 4 rocksdb: Options.keep_log_file_num: 1000
-130> 2016-06-14 04:51:54.325955 7faa48c7a8c0 4 rocksdb: Options.recycle_log_file_num: 16
-129> 2016-06-14 04:51:54.326188 7faa48c7a8c0 4 rocksdb: Options.allow_os_buffer: 1
-128> 2016-06-14 04:51:54.332241 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_reads: 0
-127> 2016-06-14 04:51:54.338238 7faa48c7a8c0 4 rocksdb: Options.allow_fallocate: 1
-126> 2016-06-14 04:51:54.338469 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_writes: 0
-125> 2016-06-14 04:51:54.338695 7faa48c7a8c0 4 rocksdb: Options.create_missing_column_families: 0
-124> 2016-06-14 04:51:54.338902 7faa48c7a8c0 4 rocksdb: Options.db_log_dir:
-123> 2016-06-14 04:51:54.339145 7faa48c7a8c0 4 rocksdb: Options.wal_dir: db
-122> 2016-06-14 04:51:54.345269 7faa48c7a8c0 4 rocksdb: Options.table_cache_numshardbits: 4
-121> 2016-06-14 04:51:54.345465 7faa48c7a8c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-120> 2016-06-14 04:51:54.345694 7faa48c7a8c0 4 rocksdb: Options.max_background_compactions: 1
-119> 2016-06-14 04:51:54.345900 7faa48c7a8c0 4 rocksdb: Options.max_subcompactions: 1
-118> 2016-06-14 04:51:54.346161 7faa48c7a8c0 4 rocksdb: Options.max_background_flushes: 1
-117> 2016-06-14 04:51:54.346385 7faa48c7a8c0 4 rocksdb: Options.WAL_ttl_seconds: 0
-116> 2016-06-14 04:51:54.352294 7faa48c7a8c0 4 rocksdb: Options.WAL_size_limit_MB: 0
-115> 2016-06-14 04:51:54.352561 7faa48c7a8c0 4 rocksdb: Options.manifest_preallocation_size: 4194304
-114> 2016-06-14 04:51:54.352762 7faa48c7a8c0 4 rocksdb: Options.allow_os_buffer: 1
-113> 2016-06-14 04:51:54.353007 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_reads: 0
-112> 2016-06-14 04:51:54.353239 7faa48c7a8c0 4 rocksdb: Options.allow_mmap_writes: 0
-111> 2016-06-14 04:51:54.353434 7faa48c7a8c0 4 rocksdb: Options.is_fd_close_on_exec: 1
-110> 2016-06-14 04:51:54.353674 7faa48c7a8c0 4 rocksdb: Options.stats_dump_period_sec: 600
-109> 2016-06-14 04:51:54.365191 7faa48c7a8c0 4 rocksdb: Options.advise_random_on_open: 1
-108> 2016-06-14 04:51:54.365421 7faa48c7a8c0 4 rocksdb: Options.db_write_buffer_size: 0d
-107> 2016-06-14 04:51:54.365618 7faa48c7a8c0 4 rocksdb: Options.access_hint_on_compaction_start: NORMAL
-106> 2016-06-14 04:51:54.365856 7faa48c7a8c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-105> 2016-06-14 04:51:54.366052 7faa48c7a8c0 4 rocksdb: Options.compaction_readahead_size: 0d
-104> 2016-06-14 04:51:54.366299 7faa48c7a8c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576d
-103> 2016-06-14 04:51:54.372240 7faa48c7a8c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576d
-102> 2016-06-14 04:51:54.372504 7faa48c7a8c0 4 rocksdb: Options.use_adaptive_mutex: 0
-101> 2016-06-14 04:51:54.372701 7faa48c7a8c0 4 rocksdb: Options.rate_limiter: (nil)
-100> 2016-06-14 04:51:54.372971 7faa48c7a8c0 4 rocksdb: Options.delete_scheduler.rate_bytes_per_sec: 0
-99> 2016-06-14 04:51:54.373192 7faa48c7a8c0 4 rocksdb: Options.bytes_per_sync: 0
-98> 2016-06-14 04:51:54.373407 7faa48c7a8c0 4 rocksdb: Options.wal_bytes_per_sync: 0
-97> 2016-06-14 04:51:54.379290 7faa48c7a8c0 4 rocksdb: Options.wal_recovery_mode: 0
-96> 2016-06-14 04:51:54.379537 7faa48c7a8c0 4 rocksdb: Options.enable_thread_tracking: 0
-95> 2016-06-14 04:51:54.379744 7faa48c7a8c0 4 rocksdb: Options.row_cache: None
-94> 2016-06-14 04:51:54.380011 7faa48c7a8c0 4 rocksdb: Options.wal_filter: None
-93> 2016-06-14 04:51:54.380236 7faa48c7a8c0 4 rocksdb: Compression algorithms supported:
-92> 2016-06-14 04:51:54.386222 7faa48c7a8c0 4 rocksdb: Snappy supported: 1
-91> 2016-06-14 04:51:54.386461 7faa48c7a8c0 4 rocksdb: Zlib supported: 1
-90> 2016-06-14 04:51:54.392176 7faa48c7a8c0 4 rocksdb: Bzip supported: 0
-89> 2016-06-14 04:51:54.392403 7faa48c7a8c0 4 rocksdb: LZ4 supported: 0
-88> 2016-06-14 04:51:54.392620 7faa48c7a8c0 4 rocksdb: Fast CRC32 supported: 0
-87> 2016-06-14 04:51:54.394593 7faa48c7a8c0 4 rocksdb: Recovering from manifest file: MANIFEST-028105

-86> 2016-06-14 04:51:54.394896 7faa48c7a8c0 4 rocksdb: --------------- Options for column family [default]:

-85> 2016-06-14 04:51:54.395152 7faa48c7a8c0 4 rocksdb: Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
-84> 2016-06-14 04:51:54.395385 7faa48c7a8c0 4 rocksdb: Options.merge_operator: None
-83> 2016-06-14 04:51:54.395596 7faa48c7a8c0 4 rocksdb: Options.compaction_filter: None
-82> 2016-06-14 04:51:54.407254 7faa48c7a8c0 4 rocksdb: Options.compaction_filter_factory: None
-81> 2016-06-14 04:51:54.418722 7faa48c7a8c0 4 rocksdb: Options.memtable_factory: SkipListFactory
-80> 2016-06-14 04:51:54.418975 7faa48c7a8c0 4 rocksdb: Options.table_factory: BlockBasedTable
-79> 2016-06-14 04:51:54.419241 7faa48c7a8c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f1bfdcc108)
cache_index_and_filter_blocks: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x55f1bfdf5840
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 0

-78> 2016-06-14 04:51:54.431426 7faa48c7a8c0 4 rocksdb: Options.write_buffer_size: 4194304
-77> 2016-06-14 04:51:54.439305 7faa48c7a8c0 4 rocksdb: Options.max_write_buffer_number: 16
-76> 2016-06-14 04:51:54.439563 7faa48c7a8c0 4 rocksdb: Options.compression: NoCompression
-75> 2016-06-14 04:51:54.439779 7faa48c7a8c0 4 rocksdb: Options.prefix_extractor: nullptr
-74> 2016-06-14 04:51:54.440019 7faa48c7a8c0 4 rocksdb: Options.num_levels: 7
-73> 2016-06-14 04:51:54.440237 7faa48c7a8c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 3
-72> 2016-06-14 04:51:54.440456 7faa48c7a8c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-71> 2016-06-14 04:51:54.440667 7faa48c7a8c0 4 rocksdb: Options.compression_opts.window_bits: -14
-70> 2016-06-14 04:51:54.440897 7faa48c7a8c0 4 rocksdb: Options.compression_opts.level: -1
-69> 2016-06-14 04:51:54.441140 7faa48c7a8c0 4 rocksdb: Options.compression_opts.strategy: 0
-68> 2016-06-14 04:51:54.441339 7faa48c7a8c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-67> 2016-06-14 04:51:54.441561 7faa48c7a8c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-66> 2016-06-14 04:51:54.441772 7faa48c7a8c0 4 rocksdb: Options.level0_stop_writes_trigger: 24
-65> 2016-06-14 04:51:54.441994 7faa48c7a8c0 4 rocksdb: Options.target_file_size_base: 2097152
-64> 2016-06-14 04:51:54.442220 7faa48c7a8c0 4 rocksdb: Options.target_file_size_multiplier: 1
-63> 2016-06-14 04:51:54.442432 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_base: 10485760
-62> 2016-06-14 04:51:54.442654 7faa48c7a8c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-61> 2016-06-14 04:51:54.442902 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10
-60> 2016-06-14 04:51:54.443142 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-59> 2016-06-14 04:51:54.443341 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-58> 2016-06-14 04:51:54.443563 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-57> 2016-06-14 04:51:54.474333 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-56> 2016-06-14 04:51:54.478289 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-55> 2016-06-14 04:51:54.478539 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-54> 2016-06-14 04:51:54.478757 7faa48c7a8c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-53> 2016-06-14 04:51:54.478983 7faa48c7a8c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-52> 2016-06-14 04:51:54.479217 7faa48c7a8c0 4 rocksdb: Options.expanded_compaction_factor: 25
-51> 2016-06-14 04:51:54.479428 7faa48c7a8c0 4 rocksdb: Options.source_compaction_factor: 1
-50> 2016-06-14 04:51:54.479663 7faa48c7a8c0 4 rocksdb: Options.max_grandparent_overlap_factor: 10
-49> 2016-06-14 04:51:54.479869 7faa48c7a8c0 4 rocksdb: Options.arena_block_size: 524288
-48> 2016-06-14 04:51:54.480091 7faa48c7a8c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 0
-47> 2016-06-14 04:51:54.480304 7faa48c7a8c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 0
-46> 2016-06-14 04:51:54.480529 7faa48c7a8c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
-45> 2016-06-14 04:51:54.480738 7faa48c7a8c0 4 rocksdb: Options.disable_auto_compactions: 0
-44> 2016-06-14 04:51:54.480960 7faa48c7a8c0 4 rocksdb: Options.filter_deletes: 0
-43> 2016-06-14 04:51:54.481184 7faa48c7a8c0 4 rocksdb: Options.verify_checksums_in_compaction: 1
-42> 2016-06-14 04:51:54.481397 7faa48c7a8c0 4 rocksdb: Options.compaction_style: 0
-41> 2016-06-14 04:51:54.481624 7faa48c7a8c0 4 rocksdb: Options.compaction_pri: 0
-40> 2016-06-14 04:51:54.481851 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-39> 2016-06-14 04:51:54.482078 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-38> 2016-06-14 04:51:54.482323 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-37> 2016-06-14 04:51:54.482543 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-36> 2016-06-14 04:51:54.482770 7faa48c7a8c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-35> 2016-06-14 04:51:54.482991 7faa48c7a8c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-34> 2016-06-14 04:51:54.495245 7faa48c7a8c0 4 rocksdb: Options.table_properties_collectors:
-33> 2016-06-14 04:51:54.522550 7faa48c7a8c0 4 rocksdb: Options.inplace_update_support: 0
-32> 2016-06-14 04:51:54.522902 7faa48c7a8c0 4 rocksdb: Options.inplace_update_num_locks: 10000
-31> 2016-06-14 04:51:54.523138 7faa48c7a8c0 4 rocksdb: Options.min_partial_merge_operands: 2
-30> 2016-06-14 04:51:54.523366 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_bits: 0
-29> 2016-06-14 04:51:54.523595 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_probes: 6
-28> 2016-06-14 04:51:54.523817 7faa48c7a8c0 4 rocksdb: Options.memtable_prefix_bloom_huge_page_tlb_size: 0
-27> 2016-06-14 04:51:54.524034 7faa48c7a8c0 4 rocksdb: Options.bloom_locality: 0
-26> 2016-06-14 04:51:54.524260 7faa48c7a8c0 4 rocksdb: Options.max_successive_merges: 0
-25> 2016-06-14 04:51:54.524481 7faa48c7a8c0 4 rocksdb: Options.optimize_fllters_for_hits: 0
-24> 2016-06-14 04:51:54.524704 7faa48c7a8c0 4 rocksdb: Options.paranoid_file_checks: 0
-23> 2016-06-14 04:51:54.524923 7faa48c7a8c0 4 rocksdb: Options.compaction_measure_io_stats: 0
-22> 2016-06-14 04:51:54.532672 7faa48c7a8c0 4 rocksdb: Recovered from manifest file:db/MANIFEST-028105 succeeded,manifest_file_number is 28105, next_file_number is 28107, last_sequence is 3262218, log_number is 0,prev_log_number is 0,max_column_family is 0

-21> 2016-06-14 04:51:54.552661 7faa48c7a8c0 4 rocksdb: Column family [default] (ID 0), log number is 28104

-20> 2016-06-14 04:51:54.566063 7faa48c7a8c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1465879914566054, "job": 1, "event": "recovery_started", "log_files": [28106]}
-19> 2016-06-14 04:51:54.582565 7faa48c7a8c0 4 rocksdb: Recovering log #28106 mode 0 skip-recovery 0
-18> 2016-06-14 04:51:54.582831 7faa48c7a8c0 4 rocksdb: Creating manifest 28108

-17> 2016-06-14 04:51:54.583887 7faa48c7a8c0 4 rocksdb: Deleting manifest 28105 current manifest 28108

-16> 2016-06-14 04:51:54.584208 7faa48c7a8c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1465879914584200, "job": 1, "event": "recovery_finished"}
-15> 2016-06-14 04:51:54.642554 7faa48c7a8c0 5 rocksdb: [JOB 2] Delete db//028106.log type=0 #28106 -- OK

-14> 2016-06-14 04:51:54.681961 7faa48c7a8c0 4 rocksdb: DB pointer 0x55f1c055a000
-13> 2016-06-14 04:51:54.709583 7faa48c7a8c0 1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=16
-12> 2016-06-14 04:51:54.709870 7faa48c7a8c0 1 freelist init prefix B
-11> 2016-06-14 04:51:54.747064 7faa48c7a8c0 2 osd.3 0 boot
-10> 2016-06-14 04:51:54.774510 7faa48c7a8c0 1 <cls> cls/version/cls_version.cc:228: Loaded version class!
-9> 2016-06-14 04:51:54.774695 7faa48c7a8c0 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
-8> 2016-06-14 04:51:54.921069 7faa48c7a8c0 1 <cls> cls/refcount/cls_refcount.cc:232: Loaded refcount class!
-7> 2016-06-14 04:51:54.964903 7faa48c7a8c0 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-6> 2016-06-14 04:51:54.965054 7faa48c7a8c0 1 <cls> cls/log/cls_log.cc:317: Loaded log class!
-5> 2016-06-14 04:51:54.965444 7faa48c7a8c0 1 <cls> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class!
-4> 2016-06-14 04:51:54.967900 7faa48c7a8c0 1 <cls> cls/rgw/cls_rgw.cc:3207: Loaded rgw class!
-3> 2016-06-14 04:51:54.980307 7faa48c7a8c0 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-2> 2016-06-14 04:51:54.980440 7faa48c7a8c0 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
-1> 2016-06-14 04:51:54.991720 7faa48c7a8c0 1 <cls> cls/user/cls_user.cc:375: Loaded user class!
0> 2016-06-14 04:51:55.017680 7faa48c7a8c0 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7faa48c7a8c0 time 2016-06-14 04:51:55.000137
osd/OSD.h: 885: FAILED assert(ret)

ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f1b486fd60]
2: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
3: (OSD::init()+0x1f91) [0x55f1b419b391]
4: (main()+0x2ea5) [0x55f1b410cdd5]
5: (__libc_start_main()+0xf0) [0x7faa45abf830]
6: (_start()+0x29) [0x55f1b414e469]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
*** Caught signal (Aborted) **
in thread 7faa48c7a8c0 thread_name:ceph-osd
ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (()+0x96ab3e) [0x55f1b476bb3e]
2: (()+0x113d0) [0x7faa47b343d0]
3: (gsignal()+0x38) [0x7faa45ad4418]
4: (abort()+0x16a) [0x7faa45ad601a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55f1b486ff4b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
7: (OSD::init()+0x1f91) [0x55f1b419b391]
8: (main()+0x2ea5) [0x55f1b410cdd5]
9: (__libc_start_main()+0xf0) [0x7faa45abf830]
10: (_start()+0x29) [0x55f1b414e469]
2016-06-14 04:51:55.071823 7faa48c7a8c0 -1 *** Caught signal (Aborted) **
in thread 7faa48c7a8c0 thread_name:ceph-osd

ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (()+0x96ab3e) [0x55f1b476bb3e]
2: (()+0x113d0) [0x7faa47b343d0]
3: (gsignal()+0x38) [0x7faa45ad4418]
4: (abort()+0x16a) [0x7faa45ad601a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55f1b486ff4b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
7: (OSD::init()+0x1f91) [0x55f1b419b391]
8: (main()+0x2ea5) [0x55f1b410cdd5]
9: (__libc_start_main()+0xf0) [0x7faa45abf830]
10: (_start()+0x29) [0x55f1b414e469]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2016-06-14 04:51:55.071823 7faa48c7a8c0 -1 *** Caught signal (Aborted) **
in thread 7faa48c7a8c0 thread_name:ceph-osd

ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (()+0x96ab3e) [0x55f1b476bb3e]
2: (()+0x113d0) [0x7faa47b343d0]
3: (gsignal()+0x38) [0x7faa45ad4418]
4: (abort()+0x16a) [0x7faa45ad601a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55f1b486ff4b]
6: (OSDService::get_map(unsigned int)+0x5d) [0x55f1b41eb39d]
7: (OSD::init()+0x1f91) [0x55f1b419b391]
8: (main()+0x2ea5) [0x55f1b410cdd5]
9: (__libc_start_main()+0xf0) [0x7faa45abf830]
10: (_start()+0x29) [0x55f1b414e469]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
(2-2/3)