Project

General

Profile

Bug #20933

All mon nodes down when i use ceph-disk prepare a new osd.

Added by chuan jiang about 2 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
Start date:
08/07/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)

when "ceph-disk prepare --bluestore /dev/vdd --osd-uuid `uuidgen`" completed with successful, it occurred.
There are two mon nodes in my environment.And all of it comes down with restart failed.

Here is some log.

2017-08-07 15:23:21.903666 7f0694594bc0 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-07 15:23:21.903667 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-07 15:23:21.903668 7f0694594bc0 4 rocksdb: Options.compression: NoCompression
2017-08-07 15:23:21.903669 7f0694594bc0 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-07 15:23:21.903670 7f0694594bc0 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-07 15:23:21.903671 7f0694594bc0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-07 15:23:21.903676 7f0694594bc0 4 rocksdb: Options.num_levels: 7
2017-08-07 15:23:21.903677 7f0694594bc0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-07 15:23:21.903678 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-07 15:23:21.903684 7f0694594bc0 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-07 15:23:21.903685 7f0694594bc0 4 rocksdb: Options.compression_opts.level: -1
2017-08-07 15:23:21.903686 7f0694594bc0 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-07 15:23:21.903687 7f0694594bc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-07 15:23:21.903688 7f0694594bc0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-07 15:23:21.903690 7f0694594bc0 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-07 15:23:21.903691 7f0694594bc0 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-07 15:23:21.903692 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-07 15:23:21.903693 7f0694594bc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-07 15:23:21.903694 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-07 15:23:21.903696 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl0: 1
2017-08-07 15:23:21.903697 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl1: 1
2017-08-07 15:23:21.903698 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl2: 1
2017-08-07 15:23:21.903699 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl3: 1
2017-08-07 15:23:21.903700 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl4: 1
2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl5: 1
2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl6: 1
2017-08-07 15:23:21.903702 7f0694594bc0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-07 15:23:21.903703 7f0694594bc0 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-07 15:23:21.903704 7f0694594bc0 4 rocksdb: Options.arena_block_size: 4194304
2017-08-07 15:23:21.903705 7f0694594bc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-07 15:23:21.903706 7f0694594bc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-07 15:23:21.903709 7f0694594bc0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-07 15:23:21.903710 7f0694594bc0 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-07 15:23:21.903711 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-07 15:23:21.903712 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-07 15:23:21.903713 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-07 15:23:21.903714 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-07 15:23:21.903719 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-07 15:23:21.903720 7f0694594bc0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-07 15:23:21.903721 7f0694594bc0 4 rocksdb: Options.table_properties_collectors:
2017-08-07 15:23:21.903722 7f0694594bc0 4 rocksdb: Options.inplace_update_support: 0
2017-08-07 15:23:21.903723 7f0694594bc0 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-07 15:23:21.903728 7f0694594bc0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-07 15:23:21.903729 7f0694594bc0 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.bloom_locality: 0
2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.max_successive_merges: 0
2017-08-07 15:23:21.903731 7f0694594bc0 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-07 15:23:21.903732 7f0694594bc0 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.force_consistency_checks: 0
2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-07 15:23:21.904837 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph1-cbm/store.db/MANIFEST-019238 succeeded,manifest_file_number is 19238, next_file_number is 19240, last_sequence is 6357165, log_number is 0,prev_log_number is 0,max_column_family is 0

2017-08-07 15:23:21.904852 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 19237

2017-08-07 15:23:21.904915 7f0694594bc0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601904907, "job": 1, "event": "recovery_started", "log_files": [19239]}
2017-08-07 15:23:21.904925 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #19239 mode 2
2017-08-07 15:23:21.905043 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 19241

2017-08-07 15:23:21.908533 7f0694594bc0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601908528, "job": 1, "event": "recovery_finished"}
2017-08-07 15:23:21.912124 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f069da64000
2017-08-07 15:23:21.912878 7f0694594bc0 0 starting mon.ceph1-cbm rank 0 at 10.10.20.203:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph1-cbm fsid f819b005-e560-4281-9eed-263340833437
2017-08-07 15:23:21.913251 7f0694594bc0 1 mon.ceph1-cbm@-1(probing) e1 preinit fsid f819b005-e560-4281-9eed-263340833437
2017-08-07 15:23:21.913415 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).mds e0 Unable to load 'last_metadata'
2017-08-07 15:23:21.913464 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-07 15:23:21.913472 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2017-08-07 15:23:21.913521 7f0694594bc0 0 mon.ceph1-cbm@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: -1

No filesystems configured

2017-08-07 15:23:21.916184 7f0694594bc0 -1 /build/ceph-12.1.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7f0694594bc0 time 2017-08-07 15:23:21.913677
/build/ceph-12.1.0/src/mon/OSDMonitor.cc: 2977: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f0693d65d1e]
2: (()+0x4a19cb) [0x7f0693c7f9cb]
3: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
4: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
5: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
6: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
7: (Monitor::preinit()+0x973) [0x7f0693b6af73]
8: (main()+0x36c6) [0x7f0693aa14d6]
9: (__libc_start_main()+0xf5) [0x7f06911c2f45]
10: (()+0x36177e) [0x7f0693b3f77e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-194> 2017-08-07 15:23:21.891106 7f0694594bc0 5 asok(0x7f069d916000) register_command perfcounters_dump hook 0x7f069d7c2180
-193> 2017-08-07 15:23:21.891114 7f0694594bc0 5 asok(0x7f069d916000) register_command 1 hook 0x7f069d7c2180
-192> 2017-08-07 15:23:21.891117 7f0694594bc0 5 asok(0x7f069d916000) register_command perf dump hook 0x7f069d7c2180
-191> 2017-08-07 15:23:21.891120 7f0694594bc0 5 asok(0x7f069d916000) register_command perfcounters_schema hook 0x7f069d7c2180
-190> 2017-08-07 15:23:21.891122 7f0694594bc0 5 asok(0x7f069d916000) register_command perf histogram dump hook 0x7f069d7c2180
-189> 2017-08-07 15:23:21.891123 7f0694594bc0 5 asok(0x7f069d916000) register_command 2 hook 0x7f069d7c2180
-188> 2017-08-07 15:23:21.891127 7f0694594bc0 5 asok(0x7f069d916000) register_command perf schema hook 0x7f069d7c2180
-187> 2017-08-07 15:23:21.891130 7f0694594bc0 5 asok(0x7f069d916000) register_command perf histogram schema hook 0x7f069d7c2180
-186> 2017-08-07 15:23:21.891135 7f0694594bc0 5 asok(0x7f069d916000) register_command perf reset hook 0x7f069d7c2180
-185> 2017-08-07 15:23:21.891137 7f0694594bc0 5 asok(0x7f069d916000) register_command config show hook 0x7f069d7c2180
-184> 2017-08-07 15:23:21.891140 7f0694594bc0 5 asok(0x7f069d916000) register_command config set hook 0x7f069d7c2180
-183> 2017-08-07 15:23:21.891143 7f0694594bc0 5 asok(0x7f069d916000) register_command config get hook 0x7f069d7c2180
-182> 2017-08-07 15:23:21.891152 7f0694594bc0 5 asok(0x7f069d916000) register_command config diff hook 0x7f069d7c2180
-181> 2017-08-07 15:23:21.891169 7f0694594bc0 5 asok(0x7f069d916000) register_command config diff get hook 0x7f069d7c2180
-180> 2017-08-07 15:23:21.891172 7f0694594bc0 5 asok(0x7f069d916000) register_command log flush hook 0x7f069d7c2180
-179> 2017-08-07 15:23:21.891174 7f0694594bc0 5 asok(0x7f069d916000) register_command log dump hook 0x7f069d7c2180
-178> 2017-08-07 15:23:21.891176 7f0694594bc0 5 asok(0x7f069d916000) register_command log reopen hook 0x7f069d7c2180
-177> 2017-08-07 15:23:21.891184 7f0694594bc0 5 asok(0x7f069d916000) register_command dump_mempools hook 0x7f069d8b7da8
-176> 2017-08-07 15:23:21.896091 7f0694594bc0 0 set uid:gid to 64045:64045 (ceph:ceph)
-175> 2017-08-07 15:23:21.896117 7f0694594bc0 0 ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev), process (unknown), pid 15523
-174> 2017-08-07 15:23:21.896180 7f0694594bc0 0 pidfile_write: ignore empty --pid-file
-173> 2017-08-07 15:23:21.897640 7f0694594bc0 5 asok(0x7f069d916000) init /var/run/ceph/ceph-mon.ceph1-cbm.asok
-172> 2017-08-07 15:23:21.897657 7f0694594bc0 5 asok(0x7f069d916000) bind_and_listen /var/run/ceph/ceph-mon.ceph1-cbm.asok
-171> 2017-08-07 15:23:21.897712 7f0694594bc0 5 asok(0x7f069d916000) register_command 0 hook 0x7f069d7c0150
-170> 2017-08-07 15:23:21.897722 7f0694594bc0 5 asok(0x7f069d916000) register_command version hook 0x7f069d7c0150
-169> 2017-08-07 15:23:21.897725 7f0694594bc0 5 asok(0x7f069d916000) register_command git_version hook 0x7f069d7c0150
-168> 2017-08-07 15:23:21.897728 7f0694594bc0 5 asok(0x7f069d916000) register_command help hook 0x7f069d7c2330
-167> 2017-08-07 15:23:21.897732 7f0694594bc0 5 asok(0x7f069d916000) register_command get_command_descriptions hook 0x7f069d7c2340
-166> 2017-08-07 15:23:21.897787 7f068e781700 5 asok(0x7f069d916000) entry start
-165> 2017-08-07 15:23:21.902984 7f0694594bc0 0 load: jerasure load: lrc load: isa
-164> 2017-08-07 15:23:21.903087 7f0694594bc0 0 set rocksdb option compression = kNoCompression
-163> 2017-08-07 15:23:21.903101 7f0694594bc0 0 set rocksdb option write_buffer_size = 33554432
-162> 2017-08-07 15:23:21.903127 7f0694594bc0 0 set rocksdb option compression = kNoCompression
-161> 2017-08-07 15:23:21.903131 7f0694594bc0 0 set rocksdb option write_buffer_size = 33554432
-160> 2017-08-07 15:23:21.903254 7f0694594bc0 4 rocksdb: RocksDB version: 5.4.0

-159> 2017-08-07 15:23:21.903267 7f0694594bc0  4 rocksdb: Git sha rocksdb_build_git_sha:0
-158> 2017-08-07 15:23:21.903269 7f0694594bc0 4 rocksdb: Compile date Jun 22 2017
-157> 2017-08-07 15:23:21.903274 7f0694594bc0 4 rocksdb: DB SUMMARY
-156> 2017-08-07 15:23:21.903312 7f0694594bc0  4 rocksdb: CURRENT file:  CURRENT
-155> 2017-08-07 15:23:21.903320 7f0694594bc0  4 rocksdb: IDENTITY file:  IDENTITY
-154> 2017-08-07 15:23:21.903323 7f0694594bc0  4 rocksdb: MANIFEST file:  MANIFEST-019238 size: 183 Bytes
-153> 2017-08-07 15:23:21.903325 7f0694594bc0  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-ceph1-cbm/store.db dir, Total Num: 2, files: 018956.sst 018958.sst
-152> 2017-08-07 15:23:21.903328 7f0694594bc0  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-ceph1-cbm/store.db: 019239.log size: 0 ;
-151> 2017-08-07 15:23:21.903330 7f0694594bc0  4 rocksdb:                         Options.error_if_exists: 0
-150> 2017-08-07 15:23:21.903331 7f0694594bc0 4 rocksdb: Options.create_if_missing: 0
-149> 2017-08-07 15:23:21.903332 7f0694594bc0 4 rocksdb: Options.paranoid_checks: 1
-148> 2017-08-07 15:23:21.903332 7f0694594bc0 4 rocksdb: Options.env: 0x7f069c7fc960
-147> 2017-08-07 15:23:21.903333 7f0694594bc0 4 rocksdb: Options.info_log: 0x7f069d96b7a0
-146> 2017-08-07 15:23:21.903334 7f0694594bc0 4 rocksdb: Options.max_open_files: -1
-145> 2017-08-07 15:23:21.903335 7f0694594bc0 4 rocksdb: Options.max_file_opening_threads: 16
-144> 2017-08-07 15:23:21.903336 7f0694594bc0 4 rocksdb: Options.use_fsync: 0
-143> 2017-08-07 15:23:21.903337 7f0694594bc0 4 rocksdb: Options.max_log_file_size: 0
-142> 2017-08-07 15:23:21.903338 7f0694594bc0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-141> 2017-08-07 15:23:21.903339 7f0694594bc0 4 rocksdb: Options.log_file_time_to_roll: 0
-140> 2017-08-07 15:23:21.903340 7f0694594bc0 4 rocksdb: Options.keep_log_file_num: 1000
-139> 2017-08-07 15:23:21.903340 7f0694594bc0 4 rocksdb: Options.recycle_log_file_num: 0
-138> 2017-08-07 15:23:21.903341 7f0694594bc0 4 rocksdb: Options.allow_fallocate: 1
-137> 2017-08-07 15:23:21.903342 7f0694594bc0 4 rocksdb: Options.allow_mmap_reads: 0
-136> 2017-08-07 15:23:21.903342 7f0694594bc0 4 rocksdb: Options.allow_mmap_writes: 0
-135> 2017-08-07 15:23:21.903343 7f0694594bc0 4 rocksdb: Options.use_direct_reads: 0
-134> 2017-08-07 15:23:21.903344 7f0694594bc0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
-133> 2017-08-07 15:23:21.903345 7f0694594bc0 4 rocksdb: Options.create_missing_column_families: 0
-132> 2017-08-07 15:23:21.903346 7f0694594bc0 4 rocksdb: Options.db_log_dir:
-131> 2017-08-07 15:23:21.903347 7f0694594bc0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-ceph1-cbm/store.db
-130> 2017-08-07 15:23:21.903348 7f0694594bc0 4 rocksdb: Options.table_cache_numshardbits: 6
-129> 2017-08-07 15:23:21.903349 7f0694594bc0 4 rocksdb: Options.max_subcompactions: 1
-128> 2017-08-07 15:23:21.903350 7f0694594bc0 4 rocksdb: Options.max_background_flushes: 1
-127> 2017-08-07 15:23:21.903351 7f0694594bc0 4 rocksdb: Options.WAL_ttl_seconds: 0
-126> 2017-08-07 15:23:21.903352 7f0694594bc0 4 rocksdb: Options.WAL_size_limit_MB: 0
-125> 2017-08-07 15:23:21.903353 7f0694594bc0 4 rocksdb: Options.manifest_preallocation_size: 4194304
-124> 2017-08-07 15:23:21.903354 7f0694594bc0 4 rocksdb: Options.is_fd_close_on_exec: 1
-123> 2017-08-07 15:23:21.903355 7f0694594bc0 4 rocksdb: Options.advise_random_on_open: 1
-122> 2017-08-07 15:23:21.903356 7f0694594bc0 4 rocksdb: Options.db_write_buffer_size: 0
-121> 2017-08-07 15:23:21.903357 7f0694594bc0 4 rocksdb: Options.access_hint_on_compaction_start: 1
-120> 2017-08-07 15:23:21.903357 7f0694594bc0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-119> 2017-08-07 15:23:21.903358 7f0694594bc0 4 rocksdb: Options.compaction_readahead_size: 0
-118> 2017-08-07 15:23:21.903359 7f0694594bc0 4 rocksdb: Options.random_access_max_buffer_size: 1048576
-117> 2017-08-07 15:23:21.903360 7f0694594bc0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
-116> 2017-08-07 15:23:21.903360 7f0694594bc0 4 rocksdb: Options.use_adaptive_mutex: 0
-115> 2017-08-07 15:23:21.903361 7f0694594bc0 4 rocksdb: Options.rate_limiter: (nil)
-114> 2017-08-07 15:23:21.903362 7f0694594bc0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
-113> 2017-08-07 15:23:21.903363 7f0694594bc0 4 rocksdb: Options.bytes_per_sync: 0
-112> 2017-08-07 15:23:21.903364 7f0694594bc0 4 rocksdb: Options.wal_bytes_per_sync: 0
-111> 2017-08-07 15:23:21.903364 7f0694594bc0 4 rocksdb: Options.wal_recovery_mode: 2
-110> 2017-08-07 15:23:21.903365 7f0694594bc0 4 rocksdb: Options.enable_thread_tracking: 0
-109> 2017-08-07 15:23:21.903366 7f0694594bc0 4 rocksdb: Options.allow_concurrent_memtable_write: 1
-108> 2017-08-07 15:23:21.903367 7f0694594bc0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
-107> 2017-08-07 15:23:21.903367 7f0694594bc0 4 rocksdb: Options.write_thread_max_yield_usec: 100
-106> 2017-08-07 15:23:21.903368 7f0694594bc0 4 rocksdb: Options.write_thread_slow_yield_usec: 3
-105> 2017-08-07 15:23:21.903369 7f0694594bc0 4 rocksdb: Options.row_cache: 26843546
-104> 2017-08-07 15:23:21.903370 7f0694594bc0 4 rocksdb: Options.wal_filter: None
-103> 2017-08-07 15:23:21.903371 7f0694594bc0 4 rocksdb: Options.avoid_flush_during_recovery: 0
-102> 2017-08-07 15:23:21.903372 7f0694594bc0 4 rocksdb: Options.base_background_compactions: 1
-101> 2017-08-07 15:23:21.903373 7f0694594bc0 4 rocksdb: Options.max_background_compactions: 1
-100> 2017-08-07 15:23:21.903373 7f0694594bc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0
-99> 2017-08-07 15:23:21.903374 7f0694594bc0 4 rocksdb: Options.delayed_write_rate : 16777216
-98> 2017-08-07 15:23:21.903375 7f0694594bc0 4 rocksdb: Options.max_total_wal_size: 0
-97> 2017-08-07 15:23:21.903376 7f0694594bc0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-96> 2017-08-07 15:23:21.903377 7f0694594bc0 4 rocksdb: Options.stats_dump_period_sec: 600
-95> 2017-08-07 15:23:21.903378 7f0694594bc0 4 rocksdb: Compression algorithms supported:
-94> 2017-08-07 15:23:21.903378 7f0694594bc0 4 rocksdb: Snappy supported: 0
-93> 2017-08-07 15:23:21.903379 7f0694594bc0 4 rocksdb: Zlib supported: 0
-92> 2017-08-07 15:23:21.903380 7f0694594bc0 4 rocksdb: Bzip supported: 0
-91> 2017-08-07 15:23:21.903381 7f0694594bc0 4 rocksdb: LZ4 supported: 0
-90> 2017-08-07 15:23:21.903381 7f0694594bc0 4 rocksdb: ZSTD supported: 0
-89> 2017-08-07 15:23:21.903393 7f0694594bc0 4 rocksdb: Fast CRC32 supported: 0
-88> 2017-08-07 15:23:21.903532 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-019238
87> 2017-08-07 15:23:21.903624 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/column_family.cc:407] -------------- Options for column family [default]:
-86> 2017-08-07 15:23:21.903635 7f0694594bc0  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
-85> 2017-08-07 15:23:21.903637 7f0694594bc0 4 rocksdb: Options.merge_operator:
-84> 2017-08-07 15:23:21.903638 7f0694594bc0 4 rocksdb: Options.compaction_filter: None
-83> 2017-08-07 15:23:21.903639 7f0694594bc0 4 rocksdb: Options.compaction_filter_factory: None
-82> 2017-08-07 15:23:21.903640 7f0694594bc0 4 rocksdb: Options.memtable_factory: SkipListFactory
-81> 2017-08-07 15:23:21.903641 7f0694594bc0 4 rocksdb: Options.table_factory: BlockBasedTable
-80> 2017-08-07 15:23:21.903661 7f0694594bc0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f069d7c0158)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7f069d962a98
block_cache_name: LRUCache
block_cache_options:
capacity : 107374182
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
format_version: 2
-79> 2017-08-07 15:23:21.903666 7f0694594bc0  4 rocksdb:        Options.write_buffer_size: 33554432
-78> 2017-08-07 15:23:21.903667 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number: 2
-77> 2017-08-07 15:23:21.903668 7f0694594bc0 4 rocksdb: Options.compression: NoCompression
-76> 2017-08-07 15:23:21.903669 7f0694594bc0 4 rocksdb: Options.bottommost_compression: Disabled
-75> 2017-08-07 15:23:21.903670 7f0694594bc0 4 rocksdb: Options.prefix_extractor: nullptr
-74> 2017-08-07 15:23:21.903671 7f0694594bc0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
-73> 2017-08-07 15:23:21.903676 7f0694594bc0 4 rocksdb: Options.num_levels: 7
-72> 2017-08-07 15:23:21.903677 7f0694594bc0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
-71> 2017-08-07 15:23:21.903678 7f0694594bc0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-70> 2017-08-07 15:23:21.903684 7f0694594bc0 4 rocksdb: Options.compression_opts.window_bits: -14
-69> 2017-08-07 15:23:21.903685 7f0694594bc0 4 rocksdb: Options.compression_opts.level: -1
-68> 2017-08-07 15:23:21.903686 7f0694594bc0 4 rocksdb: Options.compression_opts.strategy: 0
-67> 2017-08-07 15:23:21.903687 7f0694594bc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
-66> 2017-08-07 15:23:21.903688 7f0694594bc0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-65> 2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-64> 2017-08-07 15:23:21.903689 7f0694594bc0 4 rocksdb: Options.level0_stop_writes_trigger: 36
-63> 2017-08-07 15:23:21.903690 7f0694594bc0 4 rocksdb: Options.target_file_size_base: 67108864
-62> 2017-08-07 15:23:21.903691 7f0694594bc0 4 rocksdb: Options.target_file_size_multiplier: 1
-61> 2017-08-07 15:23:21.903692 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456
-60> 2017-08-07 15:23:21.903693 7f0694594bc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-59> 2017-08-07 15:23:21.903694 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
-58> 2017-08-07 15:23:21.903696 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-57> 2017-08-07 15:23:21.903697 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-56> 2017-08-07 15:23:21.903698 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-55> 2017-08-07 15:23:21.903699 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-54> 2017-08-07 15:23:21.903700 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-53> 2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-52> 2017-08-07 15:23:21.903701 7f0694594bc0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-51> 2017-08-07 15:23:21.903702 7f0694594bc0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-50> 2017-08-07 15:23:21.903703 7f0694594bc0 4 rocksdb: Options.max_compaction_bytes: 1677721600
-49> 2017-08-07 15:23:21.903704 7f0694594bc0 4 rocksdb: Options.arena_block_size: 4194304
-48> 2017-08-07 15:23:21.903705 7f0694594bc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
-47> 2017-08-07 15:23:21.903706 7f0694594bc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
-46> 2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
-45> 2017-08-07 15:23:21.903707 7f0694594bc0 4 rocksdb: Options.disable_auto_compactions: 0
-44> 2017-08-07 15:23:21.903709 7f0694594bc0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
-43> 2017-08-07 15:23:21.903710 7f0694594bc0 4 rocksdb: Options.compaction_pri: kByCompensatedSize
-42> 2017-08-07 15:23:21.903711 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-41> 2017-08-07 15:23:21.903712 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-40> 2017-08-07 15:23:21.903713 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-39> 2017-08-07 15:23:21.903714 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-38> 2017-08-07 15:23:21.903719 7f0694594bc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-37> 2017-08-07 15:23:21.903720 7f0694594bc0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-36> 2017-08-07 15:23:21.903721 7f0694594bc0 4 rocksdb: Options.table_properties_collectors:
-35> 2017-08-07 15:23:21.903722 7f0694594bc0 4 rocksdb: Options.inplace_update_support: 0
-34> 2017-08-07 15:23:21.903723 7f0694594bc0 4 rocksdb: Options.inplace_update_num_locks: 10000
-33> 2017-08-07 15:23:21.903728 7f0694594bc0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-32> 2017-08-07 15:23:21.903729 7f0694594bc0 4 rocksdb: Options.memtable_huge_page_size: 0
-31> 2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.bloom_locality: 0
-30> 2017-08-07 15:23:21.903730 7f0694594bc0 4 rocksdb: Options.max_successive_merges: 0
-29> 2017-08-07 15:23:21.903731 7f0694594bc0 4 rocksdb: Options.optimize_filters_for_hits: 0
-28> 2017-08-07 15:23:21.903732 7f0694594bc0 4 rocksdb: Options.paranoid_file_checks: 0
-27> 2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.force_consistency_checks: 0
-26> 2017-08-07 15:23:21.903733 7f0694594bc0 4 rocksdb: Options.report_bg_io_stats: 0
-25> 2017-08-07 15:23:21.904837 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-ceph1-cbm/store.db/MANIFEST-019238 succeeded,manifest_file_number is 19238, next_file_number is 19240, last_sequence is 6357165, log_number is 0,prev_log_number is 0,max_column_family is 0
-24> 2017-08-07 15:23:21.904852 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 19237
-23> 2017-08-07 15:23:21.904915 7f0694594bc0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601904907, "job": 1, "event": "recovery_started", "log_files": [19239]}
-22> 2017-08-07 15:23:21.904925 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:482] Recovering log #19239 mode 2
-21> 2017-08-07 15:23:21.905043 7f0694594bc0 4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/version_set.cc:2395] Creating manifest 19241
20> 2017-08-07 15:23:21.908533 7f0694594bc0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502090601908528, "job": 1, "event": "recovery_finished"}
-19> 2017-08-07 15:23:21.908734 7f0694594bc0 5 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph1-cbm/store.db//MANIFEST-019238 type=3 #19238 -
OK
18> 2017-08-07 15:23:21.908763 7f0694594bc0  5 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-ceph1-cbm/store.db//019239.log type=0 #19239 - OK
17> 2017-08-07 15:23:21.912124 7f0694594bc0  4 rocksdb: [/build/ceph-12.1.0/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f069da64000
-16> 2017-08-07 15:23:21.912803 7f068674a700 2 Event(0x7f069da5fa80 nevent=5000 time_id=1).set_owner idx=0 owner=139666002323200
-15> 2017-08-07 15:23:21.912823 7f0686f4b700 2 Event(0x7f069da5f880 nevent=5000 time_id=1).set_owner idx=1 owner=139666010715904
-14> 2017-08-07 15:23:21.912832 7f068774c700 2 Event(0x7f069da5fe80 nevent=5000 time_id=1).set_owner idx=2 owner=139666019108608
-13> 2017-08-07 15:23:21.912878 7f0694594bc0 0 starting mon.ceph1-cbm rank 0 at 10.10.20.203:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph1-cbm fsid f819b005-e560-4281-9eed-263340833437
-12> 2017-08-07 15:23:21.912986 7f0694594bc0 1 -
10.10.20.203:6789/0 learned_addr learned my addr 10.10.20.203:6789/0
11> 2017-08-07 15:23:21.912996 7f0694594bc0 1 - 10.10.20.203:6789/0 _finish_bind bind my_inst.addr is 10.10.20.203:6789/0
-10> 2017-08-07 15:23:21.913109 7f0694594bc0 5 adding auth protocol: cephx
-9> 2017-08-07 15:23:21.913119 7f0694594bc0 5 adding auth protocol: cephx
-8> 2017-08-07 15:23:21.913135 7f0694594bc0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-7> 2017-08-07 15:23:21.913142 7f0694594bc0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-6> 2017-08-07 15:23:21.913251 7f0694594bc0 1 mon.ceph1-cbm@-1(probing) e1 preinit fsid f819b005-e560-4281-9eed-263340833437
-5> 2017-08-07 15:23:21.913415 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).mds e0 Unable to load 'last_metadata'
-4> 2017-08-07 15:23:21.913464 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
-3> 2017-08-07 15:23:21.913472 7f0694594bc0 1 mon.ceph1-cbm@-1(probing).pg v0 on_upgrade discarding in-core PGMap
-2> 2017-08-07 15:23:21.913514 7f0694594bc0 4 mon.ceph1-cbm@-1(probing).mds e1 new map
-1> 2017-08-07 15:23:21.913521 7f0694594bc0 0 mon.ceph1-cbm@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
legacy client fscid: -1

No filesystems configured

0> 2017-08-07 15:23:21.916184 7f0694594bc0 -1 /build/ceph-12.1.0/src/mon/OSDMonitor.cc: In function 'void OSDMonitor::check_pg_creates_subs()' thread 7f0694594bc0 time 2017-08-07 15:23:21.913677
/build/ceph-12.1.0/src/mon/OSDMonitor.cc: 2977: FAILED assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB)
ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f0693d65d1e]
2: (()+0x4a19cb) [0x7f0693c7f9cb]
3: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
4: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
5: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
6: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
7: (Monitor::preinit()+0x973) [0x7f0693b6af73]
8: (main()+0x36c6) [0x7f0693aa14d6]
9: (__libc_start_main()+0xf5) [0x7f06911c2f45]
10: (()+0x36177e) [0x7f0693b3f77e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph1-cbm.log
--
end dump of recent events ---
2017-08-07 15:23:21.919768 7f0694594bc0 -1 ** Caught signal (Aborted) *
in thread 7f0694594bc0 thread_name:ceph-mon

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
1: (()+0x7fc427) [0x7f0693fda427]
2: (()+0x10330) [0x7f0692b76330]
3: (gsignal()+0x37) [0x7f06911d7c37]
4: (abort()+0x148) [0x7f06911db028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f0693d65e90]
6: (()+0x4a19cb) [0x7f0693c7f9cb]
7: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
8: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
9: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
10: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
11: (Monitor::preinit()+0x973) [0x7f0693b6af73]
12: (main()+0x36c6) [0x7f0693aa14d6]
13: (__libc_start_main()+0xf5) [0x7f06911c2f45]
14: (()+0x36177e) [0x7f0693b3f77e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2017-08-07 15:23:21.919768 7f0694594bc0 -1 ** Caught signal (Aborted) *
in thread 7f0694594bc0 thread_name:ceph-mon

ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
1: (()+0x7fc427) [0x7f0693fda427]
2: (()+0x10330) [0x7f0692b76330]
3: (gsignal()+0x37) [0x7f06911d7c37]
4: (abort()+0x148) [0x7f06911db028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f0693d65e90]
6: (()+0x4a19cb) [0x7f0693c7f9cb]
7: (OSDMonitor::update_from_paxos(bool*)+0x1625) [0x7f0693ca2955]
8: (PaxosService::refresh(bool*)+0x19a) [0x7f0693c6d87a]
9: (Monitor::refresh_from_paxos(bool*)+0x173) [0x7f0693b4bce3]
10: (Monitor::init_paxos()+0x105) [0x7f0693b4c0a5]
11: (Monitor::preinit()+0x973) [0x7f0693b6af73]
12: (main()+0x36c6) [0x7f0693aa14d6]
13: (__libc_start_main()+0xf5) [0x7f06911c2f45]
14: (()+0x36177e) [0x7f0693b3f77e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph1-cbm.log
--
end dump of recent events ---

ceph-mon.log View (47.4 KB) chuan jiang, 08/07/2017 09:01 AM

ceph-mon.ceph-mon-1.log View (52 KB) Denis Zadonskii, 08/31/2017 10:57 AM


Related issues

Duplicates RADOS - Bug #20751: osd_state not updated properly during osd-reuse-id.sh Resolved 07/22/2017

History

#1 Updated by Greg Farnum about 2 years ago

  • Project changed from Ceph to RADOS
  • Category deleted (Monitor)
  • Priority changed from Normal to Urgent

Apparently this is the result of a typo: https://www.spinics.net/lists/ceph-users/msg37317.html

But I'm not sure that's been fixed in more recent releases either.

#2 Updated by Greg Farnum about 2 years ago

  • Status changed from New to Closed

Sage thinks this has been fixed ("[12:02:12] <sage> oh, it was a problem with the reusing osd ids"). Please update to the latest RC and report back if it reproduces.

#3 Updated by Greg Farnum about 2 years ago

  • Duplicates Bug #20751: osd_state not updated properly during osd-reuse-id.sh added

#4 Updated by Denis Zadonskii about 2 years ago

I think I've hit the similar issue. Occured with 12.1.2 when tried to add host / osd (ceph-deploy osd prepare --dmcrypt bck1lv:/dev/disk/by-id/wwn-0x5000039fe5d41d38) then tried to upgrade to 12.2.0 but mons still does not start.

CentOS 7 / ceph 12.2.0

Any idea how to recover?

Also available in: Atom PDF