Project

General

Profile

Actions
Copy link

Bug #20933

closed

All mon nodes down when i use ceph-disk prepare a new osd.

Added by chuan jiang over 6 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
Ceph - v12.2.0
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)

when "ceph-disk prepare --bluestore /dev/vdd --osd-uuid `uuidgen`" completed with successful, it occurred.
There are two mon nodes in my environment.And all of it comes down with restart failed.

Here is some log.

2017-08-07 15:23:21.903666 7f0694594bc0 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-07 15:23:21.903667 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-07 15:23:21.903668 7f0694594bc0 4 rocksdb: Options.compression: NoCompression
2017-08-07 15:23:21.903669 7f0694594bc0 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-07 15:23:21.903670 7f0694594bc0 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-07 15:23:21.903671 7f0694594bc0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-07 15:23:21.903676 7f0694594bc0 4 rocksdb: Options.num_levels: 7
2017-08-07 15:23:21.903677 7f0694594bc0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-07 15:23:21.903678 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-07 15:23:21.903684 7f0694594bc0 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-07 15:23:21.903685 7f0694594bc0 4 rocksdb: Options.compression_opts.level: -1
2017-08-07 15:23:21.903686 7f0694594bc0 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-07 15:23:21.903687 7f0694594bc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-07 15:23:21.903688 7f0694594bc0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-07 15:23:21.903690 7f0694594bc0 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-07 15:23:21.903691 7f0694594bc0 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-07 15:23:21.903692 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-07 15:23:21.903693 7f0694594bc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-07 15:23:21.903694 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-07 15:23:21.903696 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl0: 1
2017-08-07 15:23:21.903697 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl1: 1
2017-08-07 15:23:21.903698 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl2: 1
2017-08-07 15:23:21.903699 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl3: 1
2017-08-07 15:23:21.903700 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl4: 1
2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl5: 1
2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl6: 1
2017-08-07 15:23:21.903702 7f0694594bc0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-07 15:23:21.903703 7f0694594bc0 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-07 15:23:21.903704 7f0694594bc0 4 rocksdb: Options.arena_block_size: 4194304
2017-08-07 15:23:21.903705 7f0694594bc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-07 15:23:21.903706 7f0694594bc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-07 15:23:21.903709 7f0694594bc0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-07 15:23:21.903710 7f0694594bc0 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-07 15:23:21.903711 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-07 15:23:21.903712 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-07 15:23:21.903713 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-07 15:23:21.903714 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-07 15:23:21.903719 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-07 15:23:21.903720 7f0694594bc0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-07 15:23:21.903721 7f0694594bc0 4 rocksdb: Options.table_properties_collectors:
2017-08-07 15:23:21.903722 7f0694594bc0 4 rocksdb: Options.inplace_update_support: 0
2017-08-07 15:23:21.903723 7f0694594bc0 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-07 15:23:21.903728 7f0694594bc0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-07 15:23:21.903729 7f0694594bc0 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.bloom_locality: 0
2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.max_successive_merges: 0
2017-08-07 15:23:21.903731 7f0694594bc0 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-07 15:23:21.903732 7f0694594bc0 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.force_consistency_checks: 0
2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-07 15:23:21.904837 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph1-cbm/store.db/MANIFEST-019238 succeeded,manifest_file_number is 19238, next_file_number is 19240, last_sequence is 6357165, log_number is 0,prev_log_number is 0,max_column_family is 0

2017-08-07 15:23:21.904852 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 19237

2017-08-07 15:23:21.904915 7f0694594bc0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601904907, "job": 1, "event": "recovery_started", "log_files": [19239]}
2017-08-07 15:23:21.904925 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #19239 mode 2
2017-08-07 15:23:21.905043 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 19241

2017-08-07 15:23:21.908533 7f0694594bc0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601908528, "job": 1, "event": "recovery_finished"}
2017-08-07 15:23:21.912124 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f069da64000
2017-08-07 15:23:21.912878 7f0694594bc0 0 starting mon.ceph1-cbm rank 0 at 10.10.20.203:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph1-cbm fsid f819b005-e560-4281-9eed-263340833437
2017-08-07 15:23:21.913251 7f0694594bc0 1 mon.ceph1-cbm@-1(probing) e1 preinit fsid f819b005-e560-4281-9eed-263340833437
2017-08-07 15:23:21.913415 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).mds e0 Unable to load 'last_metadata'
2017-08-07 15:23:21.913464 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-07 15:23:21.913472 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2017-08-07 15:23:21.913521 7f0694594bc0 0 mon.ceph1-cbm@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: -1

No filesystems configured

2017-08-07 15:23:21.916184 7f0694594bc0 -1 /build/ceph-12.1.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7f0694594bc0 time 2017-08-07 15:23:21.913677
/build/ceph-12.1.0/src/mon/OSDMonitor.cc: 2977: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f0693d65d1e]
 2: (()+0x4a19cb) [0x7f0693c7f9cb]
 3: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
 4: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
 5: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
 6: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
 7: (Monitor::preinit()+0x973) [0x7f0693b6af73]
 8: (main()+0x36c6) [0x7f0693aa14d6]
 9: (__libc_start_main()+0xf5) [0x7f06911c2f45]
 10: (()+0x36177e) [0x7f0693b3f77e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-194> 2017-08-07 15:23:21.891106 7f0694594bc0 5 asok(0x7f069d916000) register_command perfcounters_dump hook 0x7f069d7c2180
-193> 2017-08-07 15:23:21.891114 7f0694594bc0 5 asok(0x7f069d916000) register_command 1 hook 0x7f069d7c2180
-192> 2017-08-07 15:23:21.891117 7f0694594bc0 5 asok(0x7f069d916000) register_command perf dump hook 0x7f069d7c2180
-191> 2017-08-07 15:23:21.891120 7f0694594bc0 5 asok(0x7f069d916000) register_command perfcounters_schema hook 0x7f069d7c2180
-190> 2017-08-07 15:23:21.891122 7f0694594bc0 5 asok(0x7f069d916000) register_command perf histogram dump hook 0x7f069d7c2180
-189> 2017-08-07 15:23:21.891123 7f0694594bc0 5 asok(0x7f069d916000) register_command 2 hook 0x7f069d7c2180
-188> 2017-08-07 15:23:21.891127 7f0694594bc0 5 asok(0x7f069d916000) register_command perf schema hook 0x7f069d7c2180
-187> 2017-08-07 15:23:21.891130 7f0694594bc0 5 asok(0x7f069d916000) register_command perf histogram schema hook 0x7f069d7c2180
-186> 2017-08-07 15:23:21.891135 7f0694594bc0 5 asok(0x7f069d916000) register_command perf reset hook 0x7f069d7c2180
-185> 2017-08-07 15:23:21.891137 7f0694594bc0 5 asok(0x7f069d916000) register_command config show hook 0x7f069d7c2180
-184> 2017-08-07 15:23:21.891140 7f0694594bc0 5 asok(0x7f069d916000) register_command config set hook 0x7f069d7c2180
-183> 2017-08-07 15:23:21.891143 7f0694594bc0 5 asok(0x7f069d916000) register_command config get hook 0x7f069d7c2180
-182> 2017-08-07 15:23:21.891152 7f0694594bc0 5 asok(0x7f069d916000) register_command config diff hook 0x7f069d7c2180
-181> 2017-08-07 15:23:21.891169 7f0694594bc0 5 asok(0x7f069d916000) register_command config diff get hook 0x7f069d7c2180
-180> 2017-08-07 15:23:21.891172 7f0694594bc0 5 asok(0x7f069d916000) register_command log flush hook 0x7f069d7c2180
-179> 2017-08-07 15:23:21.891174 7f0694594bc0 5 asok(0x7f069d916000) register_command log dump hook 0x7f069d7c2180
-178> 2017-08-07 15:23:21.891176 7f0694594bc0 5 asok(0x7f069d916000) register_command log reopen hook 0x7f069d7c2180
-177> 2017-08-07 15:23:21.891184 7f0694594bc0 5 asok(0x7f069d916000) register_command dump_mempools hook 0x7f069d8b7da8
-176> 2017-08-07 15:23:21.896091 7f0694594bc0 0 set uid:gid to 64045:64045 (ceph:ceph)
-175> 2017-08-07 15:23:21.896117 7f0694594bc0 0 ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev), process (unknown), pid 15523
-174> 2017-08-07 15:23:21.896180 7f0694594bc0 0 pidfile_write: ignore empty --pid-file
-173> 2017-08-07 15:23:21.897640 7f0694594bc0 5 asok(0x7f069d916000) init /var/run/ceph/ceph-mon.ceph1-cbm.asok
-172> 2017-08-07 15:23:21.897657 7f0694594bc0 5 asok(0x7f069d916000) bind_and_listen /var/run/ceph/ceph-mon.ceph1-cbm.asok
-171> 2017-08-07 15:23:21.897712 7f0694594bc0 5 asok(0x7f069d916000) register_command 0 hook 0x7f069d7c0150
-170> 2017-08-07 15:23:21.897722 7f0694594bc0 5 asok(0x7f069d916000) register_command version hook 0x7f069d7c0150
-169> 2017-08-07 15:23:21.897725 7f0694594bc0 5 asok(0x7f069d916000) register_command git_version hook 0x7f069d7c0150
-168> 2017-08-07 15:23:21.897728 7f0694594bc0 5 asok(0x7f069d916000) register_command help hook 0x7f069d7c2330
-167> 2017-08-07 15:23:21.897732 7f0694594bc0 5 asok(0x7f069d916000) register_command get_command_descriptions hook 0x7f069d7c2340
-166> 2017-08-07 15:23:21.897787 7f068e781700 5 asok(0x7f069d916000) entry start
-165> 2017-08-07 15:23:21.902984 7f0694594bc0 0 load: jerasure load: lrc load: isa
-164> 2017-08-07 15:23:21.903087 7f0694594bc0 0 set rocksdb option compression = kNoCompression
-163> 2017-08-07 15:23:21.903101 7f0694594bc0 0 set rocksdb option write_buffer_size = 33554432
-162> 2017-08-07 15:23:21.903127 7f0694594bc0 0 set rocksdb option compression = kNoCompression
-161> 2017-08-07 15:23:21.903131 7f0694594bc0 0 set rocksdb option write_buffer_size = 33554432
-160> 2017-08-07 15:23:21.903254 7f0694594bc0 4 rocksdb: RocksDB version: 5.4.0

-159> 2017-08-07 15:23:21.903267 7f0694594bc0  4 rocksdb: Git sha rocksdb_build_git_sha:0
  -158> 2017-08-07 15:23:21.903269 7f0694594bc0  4 rocksdb: Compile date Jun 22 2017
  -157> 2017-08-07 15:23:21.903274 7f0694594bc0  4 rocksdb: DB SUMMARY
-156> 2017-08-07 15:23:21.903312 7f0694594bc0  4 rocksdb: CURRENT file:  CURRENT
-155> 2017-08-07 15:23:21.903320 7f0694594bc0  4 rocksdb: IDENTITY file:  IDENTITY
-154> 2017-08-07 15:23:21.903323 7f0694594bc0  4 rocksdb: MANIFEST file:  MANIFEST-019238 size: 183 Bytes
-153> 2017-08-07 15:23:21.903325 7f0694594bc0  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph1-cbm/store.db dir, Total Num: 2, files: 018956.sst 018958.sst
-152> 2017-08-07 15:23:21.903328 7f0694594bc0  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph1-cbm/store.db: 019239.log size: 0 ;
-151> 2017-08-07 15:23:21.903330 7f0694594bc0  4 rocksdb:                         Options.error_if_exists: 0
  -150> 2017-08-07 15:23:21.903331 7f0694594bc0  4 rocksdb:                       Options.create_if_missing: 0
  -149> 2017-08-07 15:23:21.903332 7f0694594bc0  4 rocksdb:                         Options.paranoid_checks: 1
  -148> 2017-08-07 15:23:21.903332 7f0694594bc0  4 rocksdb:                                     Options.env: 0x7f069c7fc960
  -147> 2017-08-07 15:23:21.903333 7f0694594bc0  4 rocksdb:                                Options.info_log: 0x7f069d96b7a0
  -146> 2017-08-07 15:23:21.903334 7f0694594bc0  4 rocksdb:                          Options.max_open_files: -1
  -145> 2017-08-07 15:23:21.903335 7f0694594bc0  4 rocksdb:                Options.max_file_opening_threads: 16
  -144> 2017-08-07 15:23:21.903336 7f0694594bc0  4 rocksdb:                               Options.use_fsync: 0
  -143> 2017-08-07 15:23:21.903337 7f0694594bc0  4 rocksdb:                       Options.max_log_file_size: 0
  -142> 2017-08-07 15:23:21.903338 7f0694594bc0  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
  -141> 2017-08-07 15:23:21.903339 7f0694594bc0  4 rocksdb:                   Options.log_file_time_to_roll: 0
  -140> 2017-08-07 15:23:21.903340 7f0694594bc0  4 rocksdb:                       Options.keep_log_file_num: 1000
  -139> 2017-08-07 15:23:21.903340 7f0694594bc0  4 rocksdb:                    Options.recycle_log_file_num: 0
  -138> 2017-08-07 15:23:21.903341 7f0694594bc0  4 rocksdb:                         Options.allow_fallocate: 1
  -137> 2017-08-07 15:23:21.903342 7f0694594bc0  4 rocksdb:                        Options.allow_mmap_reads: 0
  -136> 2017-08-07 15:23:21.903342 7f0694594bc0  4 rocksdb:                       Options.allow_mmap_writes: 0
  -135> 2017-08-07 15:23:21.903343 7f0694594bc0  4 rocksdb:                        Options.use_direct_reads: 0
  -134> 2017-08-07 15:23:21.903344 7f0694594bc0  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
  -133> 2017-08-07 15:23:21.903345 7f0694594bc0  4 rocksdb:          Options.create_missing_column_families: 0
  -132> 2017-08-07 15:23:21.903346 7f0694594bc0  4 rocksdb:                              Options.db_log_dir: 
  -131> 2017-08-07 15:23:21.903347 7f0694594bc0  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-ceph1-cbm/store.db
  -130> 2017-08-07 15:23:21.903348 7f0694594bc0  4 rocksdb:                Options.table_cache_numshardbits: 6
  -129> 2017-08-07 15:23:21.903349 7f0694594bc0  4 rocksdb:                      Options.max_subcompactions: 1
  -128> 2017-08-07 15:23:21.903350 7f0694594bc0  4 rocksdb:                  Options.max_background_flushes: 1
  -127> 2017-08-07 15:23:21.903351 7f0694594bc0  4 rocksdb:                         Options.WAL_ttl_seconds: 0
  -126> 2017-08-07 15:23:21.903352 7f0694594bc0  4 rocksdb:                       Options.WAL_size_limit_MB: 0
  -125> 2017-08-07 15:23:21.903353 7f0694594bc0  4 rocksdb:             Options.manifest_preallocation_size: 4194304
  -124> 2017-08-07 15:23:21.903354 7f0694594bc0  4 rocksdb:                     Options.is_fd_close_on_exec: 1
  -123> 2017-08-07 15:23:21.903355 7f0694594bc0  4 rocksdb:                   Options.advise_random_on_open: 1
  -122> 2017-08-07 15:23:21.903356 7f0694594bc0  4 rocksdb:                    Options.db_write_buffer_size: 0
  -121> 2017-08-07 15:23:21.903357 7f0694594bc0  4 rocksdb:         Options.access_hint_on_compaction_start: 1
  -120> 2017-08-07 15:23:21.903357 7f0694594bc0  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 0
  -119> 2017-08-07 15:23:21.903358 7f0694594bc0  4 rocksdb:               Options.compaction_readahead_size: 0
  -118> 2017-08-07 15:23:21.903359 7f0694594bc0  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
  -117> 2017-08-07 15:23:21.903360 7f0694594bc0  4 rocksdb:           Options.writable_file_max_buffer_size: 1048576
  -116> 2017-08-07 15:23:21.903360 7f0694594bc0  4 rocksdb:                      Options.use_adaptive_mutex: 0
  -115> 2017-08-07 15:23:21.903361 7f0694594bc0  4 rocksdb:                            Options.rate_limiter: (nil)
  -114> 2017-08-07 15:23:21.903362 7f0694594bc0  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
  -113> 2017-08-07 15:23:21.903363 7f0694594bc0  4 rocksdb:                          Options.bytes_per_sync: 0
  -112> 2017-08-07 15:23:21.903364 7f0694594bc0  4 rocksdb:                      Options.wal_bytes_per_sync: 0
  -111> 2017-08-07 15:23:21.903364 7f0694594bc0  4 rocksdb:                       Options.wal_recovery_mode: 2
  -110> 2017-08-07 15:23:21.903365 7f0694594bc0  4 rocksdb:                  Options.enable_thread_tracking: 0
  -109> 2017-08-07 15:23:21.903366 7f0694594bc0  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
  -108> 2017-08-07 15:23:21.903367 7f0694594bc0  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
  -107> 2017-08-07 15:23:21.903367 7f0694594bc0  4 rocksdb:             Options.write_thread_max_yield_usec: 100
  -106> 2017-08-07 15:23:21.903368 7f0694594bc0  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
  -105> 2017-08-07 15:23:21.903369 7f0694594bc0  4 rocksdb:                               Options.row_cache: 26843546
  -104> 2017-08-07 15:23:21.903370 7f0694594bc0  4 rocksdb:                              Options.wal_filter: None
  -103> 2017-08-07 15:23:21.903371 7f0694594bc0  4 rocksdb:             Options.avoid_flush_during_recovery: 0
  -102> 2017-08-07 15:23:21.903372 7f0694594bc0  4 rocksdb:             Options.base_background_compactions: 1
  -101> 2017-08-07 15:23:21.903373 7f0694594bc0  4 rocksdb:             Options.max_background_compactions: 1
  -100> 2017-08-07 15:23:21.903373 7f0694594bc0  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
   -99> 2017-08-07 15:23:21.903374 7f0694594bc0  4 rocksdb:             Options.delayed_write_rate : 16777216
   -98> 2017-08-07 15:23:21.903375 7f0694594bc0  4 rocksdb:             Options.max_total_wal_size: 0
   -97> 2017-08-07 15:23:21.903376 7f0694594bc0  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
   -96> 2017-08-07 15:23:21.903377 7f0694594bc0  4 rocksdb:                   Options.stats_dump_period_sec: 600
   -95> 2017-08-07 15:23:21.903378 7f0694594bc0  4 rocksdb: Compression algorithms supported:
   -94> 2017-08-07 15:23:21.903378 7f0694594bc0  4 rocksdb:     Snappy supported: 0
   -93> 2017-08-07 15:23:21.903379 7f0694594bc0  4 rocksdb:     Zlib supported: 0
   -92> 2017-08-07 15:23:21.903380 7f0694594bc0  4 rocksdb:     Bzip supported: 0
   -91> 2017-08-07 15:23:21.903381 7f0694594bc0  4 rocksdb:     LZ4 supported: 0
   -90> 2017-08-07 15:23:21.903381 7f0694594bc0  4 rocksdb:     ZSTD supported: 0
   -89> 2017-08-07 15:23:21.903393 7f0694594bc0  4 rocksdb: Fast CRC32 supported: 0
   -88> 2017-08-07 15:23:21.903532 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-019238
87> 2017-08-07 15:23:21.903624 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/column_family.cc:407] -------------- Options for column family [default]:
-86> 2017-08-07 15:23:21.903635 7f0694594bc0  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
   -85> 2017-08-07 15:23:21.903637 7f0694594bc0  4 rocksdb:           Options.merge_operator: 
   -84> 2017-08-07 15:23:21.903638 7f0694594bc0  4 rocksdb:        Options.compaction_filter: None
   -83> 2017-08-07 15:23:21.903639 7f0694594bc0  4 rocksdb:        Options.compaction_filter_factory: None
   -82> 2017-08-07 15:23:21.903640 7f0694594bc0  4 rocksdb:         Options.memtable_factory: SkipListFactory
   -81> 2017-08-07 15:23:21.903641 7f0694594bc0  4 rocksdb:            Options.table_factory: BlockBasedTable
   -80> 2017-08-07 15:23:21.903661 7f0694594bc0  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f069d7c0158)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f069d962a98
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 107374182
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  format_version: 2
-79> 2017-08-07 15:23:21.903666 7f0694594bc0  4 rocksdb:        Options.write_buffer_size: 33554432
   -78> 2017-08-07 15:23:21.903667 7f0694594bc0  4 rocksdb:  Options.max_write_buffer_number: 2
   -77> 2017-08-07 15:23:21.903668 7f0694594bc0  4 rocksdb:          Options.compression: NoCompression
   -76> 2017-08-07 15:23:21.903669 7f0694594bc0  4 rocksdb:                  Options.bottommost_compression: Disabled
   -75> 2017-08-07 15:23:21.903670 7f0694594bc0  4 rocksdb:       Options.prefix_extractor: nullptr
   -74> 2017-08-07 15:23:21.903671 7f0694594bc0  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
   -73> 2017-08-07 15:23:21.903676 7f0694594bc0  4 rocksdb:             Options.num_levels: 7
   -72> 2017-08-07 15:23:21.903677 7f0694594bc0  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
   -71> 2017-08-07 15:23:21.903678 7f0694594bc0  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
   -70> 2017-08-07 15:23:21.903684 7f0694594bc0  4 rocksdb:            Options.compression_opts.window_bits: -14
   -69> 2017-08-07 15:23:21.903685 7f0694594bc0  4 rocksdb:                  Options.compression_opts.level: -1
   -68> 2017-08-07 15:23:21.903686 7f0694594bc0  4 rocksdb:               Options.compression_opts.strategy: 0
   -67> 2017-08-07 15:23:21.903687 7f0694594bc0  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
   -66> 2017-08-07 15:23:21.903688 7f0694594bc0  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
   -65> 2017-08-07 15:23:21.903689 7f0694594bc0  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
   -64> 2017-08-07 15:23:21.903689 7f0694594bc0  4 rocksdb:              Options.level0_stop_writes_trigger: 36
   -63> 2017-08-07 15:23:21.903690 7f0694594bc0  4 rocksdb:                   Options.target_file_size_base: 67108864
   -62> 2017-08-07 15:23:21.903691 7f0694594bc0  4 rocksdb:             Options.target_file_size_multiplier: 1
   -61> 2017-08-07 15:23:21.903692 7f0694594bc0  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
   -60> 2017-08-07 15:23:21.903693 7f0694594bc0  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
   -59> 2017-08-07 15:23:21.903694 7f0694594bc0  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
   -58> 2017-08-07 15:23:21.903696 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
   -57> 2017-08-07 15:23:21.903697 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
   -56> 2017-08-07 15:23:21.903698 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
   -55> 2017-08-07 15:23:21.903699 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
   -54> 2017-08-07 15:23:21.903700 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
   -53> 2017-08-07 15:23:21.903701 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
   -52> 2017-08-07 15:23:21.903701 7f0694594bc0  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
   -51> 2017-08-07 15:23:21.903702 7f0694594bc0  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
   -50> 2017-08-07 15:23:21.903703 7f0694594bc0  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
   -49> 2017-08-07 15:23:21.903704 7f0694594bc0  4 rocksdb:                        Options.arena_block_size: 4194304
   -48> 2017-08-07 15:23:21.903705 7f0694594bc0  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
   -47> 2017-08-07 15:23:21.903706 7f0694594bc0  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
   -46> 2017-08-07 15:23:21.903707 7f0694594bc0  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
   -45> 2017-08-07 15:23:21.903707 7f0694594bc0  4 rocksdb:                Options.disable_auto_compactions: 0
   -44> 2017-08-07 15:23:21.903709 7f0694594bc0  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
   -43> 2017-08-07 15:23:21.903710 7f0694594bc0  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
   -42> 2017-08-07 15:23:21.903711 7f0694594bc0  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
   -41> 2017-08-07 15:23:21.903712 7f0694594bc0  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
   -40> 2017-08-07 15:23:21.903713 7f0694594bc0  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
   -39> 2017-08-07 15:23:21.903714 7f0694594bc0  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
   -38> 2017-08-07 15:23:21.903719 7f0694594bc0  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
   -37> 2017-08-07 15:23:21.903720 7f0694594bc0  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
   -36> 2017-08-07 15:23:21.903721 7f0694594bc0  4 rocksdb:                   Options.table_properties_collectors: 
   -35> 2017-08-07 15:23:21.903722 7f0694594bc0  4 rocksdb:                   Options.inplace_update_support: 0
   -34> 2017-08-07 15:23:21.903723 7f0694594bc0  4 rocksdb:                 Options.inplace_update_num_locks: 10000
   -33> 2017-08-07 15:23:21.903728 7f0694594bc0  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
   -32> 2017-08-07 15:23:21.903729 7f0694594bc0  4 rocksdb:   Options.memtable_huge_page_size: 0
   -31> 2017-08-07 15:23:21.903730 7f0694594bc0  4 rocksdb:                           Options.bloom_locality: 0
   -30> 2017-08-07 15:23:21.903730 7f0694594bc0  4 rocksdb:                    Options.max_successive_merges: 0
   -29> 2017-08-07 15:23:21.903731 7f0694594bc0  4 rocksdb:                Options.optimize_filters_for_hits: 0
   -28> 2017-08-07 15:23:21.903732 7f0694594bc0  4 rocksdb:                Options.paranoid_file_checks: 0
   -27> 2017-08-07 15:23:21.903733 7f0694594bc0  4 rocksdb:                Options.force_consistency_checks: 0
   -26> 2017-08-07 15:23:21.903733 7f0694594bc0  4 rocksdb:                Options.report_bg_io_stats: 0
   -25> 2017-08-07 15:23:21.904837 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph1-cbm/store.db/MANIFEST-019238 succeeded,manifest_file_number is 19238, next_file_number is 19240, last_sequence is 6357165, log_number is 0,prev_log_number is 0,max_column_family is 0
-24> 2017-08-07 15:23:21.904852 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 19237
-23> 2017-08-07 15:23:21.904915 7f0694594bc0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601904907, "job": 1, "event": "recovery_started", "log_files": [19239]}
   -22> 2017-08-07 15:23:21.904925 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #19239 mode 2
   -21> 2017-08-07 15:23:21.905043 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 19241
20> 2017-08-07 15:23:21.908533 7f0694594bc0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601908528, "job": 1, "event": "recovery_finished"}
   -19> 2017-08-07 15:23:21.908734 7f0694594bc0  5 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph1-cbm/store.db//MANIFEST-019238 type=3 #19238 - OK
18> 2017-08-07 15:23:21.908763 7f0694594bc0  5 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph1-cbm/store.db//019239.log type=0 #19239 - OK
17> 2017-08-07 15:23:21.912124 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f069da64000
   -16> 2017-08-07 15:23:21.912803 7f068674a700  2 Event(0x7f069da5fa80 nevent=5000 time_id=1).set_owner idx=0 owner=139666002323200
   -15> 2017-08-07 15:23:21.912823 7f0686f4b700  2 Event(0x7f069da5f880 nevent=5000 time_id=1).set_owner idx=1 owner=139666010715904
   -14> 2017-08-07 15:23:21.912832 7f068774c700  2 Event(0x7f069da5fe80 nevent=5000 time_id=1).set_owner idx=2 owner=139666019108608
   -13> 2017-08-07 15:23:21.912878 7f0694594bc0  0 starting mon.ceph1-cbm rank 0 at 10.10.20.203:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph1-cbm fsid f819b005-e560-4281-9eed-263340833437
   -12> 2017-08-07 15:23:21.912986 7f0694594bc0  1 - 10.10.20.203:6789/0 learned_addr learned my addr 10.10.20.203:6789/0
   11> 2017-08-07 15:23:21.912996 7f0694594bc0  1 - 10.10.20.203:6789/0 _finish_bind bind my_inst.addr is 10.10.20.203:6789/0
   -10> 2017-08-07 15:23:21.913109 7f0694594bc0  5 adding auth protocol: cephx
    -9> 2017-08-07 15:23:21.913119 7f0694594bc0  5 adding auth protocol: cephx
    -8> 2017-08-07 15:23:21.913135 7f0694594bc0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
    -7> 2017-08-07 15:23:21.913142 7f0694594bc0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
    -6> 2017-08-07 15:23:21.913251 7f0694594bc0  1 mon.ceph1-cbm@-1(probing) e1 preinit fsid f819b005-e560-4281-9eed-263340833437
    -5> 2017-08-07 15:23:21.913415 7f0694594bc0  1 mon.ceph1-cbm@-1(probing).mds e0 Unable to load 'last_metadata'
    -4> 2017-08-07 15:23:21.913464 7f0694594bc0  1 mon.ceph1-cbm@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
    -3> 2017-08-07 15:23:21.913472 7f0694594bc0  1 mon.ceph1-cbm@-1(probing).pg v0 on_upgrade discarding in-core PGMap
    -2> 2017-08-07 15:23:21.913514 7f0694594bc0  4 mon.ceph1-cbm@-1(probing).mds e1 new map
    -1> 2017-08-07 15:23:21.913521 7f0694594bc0  0 mon.ceph1-cbm@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: -1

No filesystems configured

0> 2017-08-07 15:23:21.916184 7f0694594bc0 -1 /build/ceph-12.1.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7f0694594bc0 time 2017-08-07 15:23:21.913677
/build/ceph-12.1.0/src/mon/OSDMonitor.cc: 2977: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)
ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f0693d65d1e]
 2: (()+0x4a19cb) [0x7f0693c7f9cb]
 3: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
 4: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
 5: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
 6: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
 7: (Monitor::preinit()+0x973) [0x7f0693b6af73]
 8: (main()+0x36c6) [0x7f0693aa14d6]
 9: (__libc_start_main()+0xf5) [0x7f06911c2f45]
 10: (()+0x36177e) [0x7f0693b3f77e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph1-cbm.log
-- end dump of recent events ---
2017-08-07 15:23:21.919768 7f0694594bc0 -1 ** Caught signal (Aborted) *
in thread 7f0694594bc0 thread_name:ceph-mon

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
 1: (()+0x7fc427) [0x7f0693fda427]
 2: (()+0x10330) [0x7f0692b76330]
 3: (gsignal()+0x37) [0x7f06911d7c37]
 4: (abort()+0x148) [0x7f06911db028]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f0693d65e90]
 6: (()+0x4a19cb) [0x7f0693c7f9cb]
 7: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
 8: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
 9: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
 10: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
 11: (Monitor::preinit()+0x973) [0x7f0693b6af73]
 12: (main()+0x36c6) [0x7f0693aa14d6]
 13: (__libc_start_main()+0xf5) [0x7f06911c2f45]
 14: (()+0x36177e) [0x7f0693b3f77e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2017-08-07 15:23:21.919768 7f0694594bc0 -1 ** Caught signal (Aborted) *
in thread 7f0694594bc0 thread_name:ceph-mon

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
 1: (()+0x7fc427) [0x7f0693fda427]
 2: (()+0x10330) [0x7f0692b76330]
 3: (gsignal()+0x37) [0x7f06911d7c37]
 4: (abort()+0x148) [0x7f06911db028]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f0693d65e90]
 6: (()+0x4a19cb) [0x7f0693c7f9cb]
 7: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
 8: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
 9: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
 10: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
 11: (Monitor::preinit()+0x973) [0x7f0693b6af73]
 12: (main()+0x36c6) [0x7f0693aa14d6]
 13: (__libc_start_main()+0xf5) [0x7f06911c2f45]
 14: (()+0x36177e) [0x7f0693b3f77e]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph1-cbm.log
-- end dump of recent events ---

Files

Download all files
ceph-mon.log (47.4 KB) ceph-mon.log chuan jiang, 08/07/2017 09:01 AM
ceph-mon.ceph-mon-1.log (52 KB) ceph-mon.ceph-mon-1.log Denis Zadonskii, 08/31/2017 10:57 AM

Related issues 1 (0 open — 1 closed)

Is duplicate of RADOS - Bug #20751: osd_state not updated properly during osd-reuse-id.shResolvedSage Weil07/22/2017

Actions
Actions
Copy link
 #1

Updated by Greg Farnum over 6 years ago

  • Project changed from Ceph to RADOS
  • Category deleted (Monitor)
  • Priority changed from Normal to Urgent

Apparently this is the result of a typo: https://www.spinics.net/lists/ceph-users/msg37317.html

But I'm not sure that's been fixed in more recent releases either.

Actions
Copy link
 #2

Updated by Greg Farnum over 6 years ago

  • Status changed from New to Closed

Sage thinks this has been fixed ("[12:02:12] <sage> oh, it was a problem with the reusing osd ids"). Please update to the latest RC and report back if it reproduces.

Actions
Copy link
 #3

Updated by Greg Farnum over 6 years ago

  • Is duplicate of Bug #20751: osd_state not updated properly during osd-reuse-id.sh added
Actions
Copy link
 #4

Updated by Denis Zadonskii over 6 years ago

I think I've hit the similar issue. Occured with 12.1.2 when tried to add host / osd (ceph-deploy osd prepare --dmcrypt bck1lv:/dev/disk/by-id/wwn-0x5000039fe5d41d38) then tried to upgrade to 12.2.0 but mons still does not start.

CentOS 7 / ceph 12.2.0

Any idea how to recover?

Actions
Copy link

Also available in: Atom PDF