2019-03-02 18:53:17.948283 7fe3640c4ec0 0 set uid:gid to 1001:1001 (ceph:ceph) 2019-03-02 18:53:17.948303 7fe3640c4ec0 0 ceph version 12.2.5-2-ge988fb6 (e988fb6c5457b9dd6f1e11ce1b5fd58b4c3a828c) luminous (stable), process (unknown), pid 3016783 2019-03-02 18:53:17.948362 7fe3640c4ec0 0 pidfile_write: ignore empty --pid-file 2019-03-02 18:53:17.980882 7fe3640c4ec0 0 load: jerasure load: lrc load: isa 2019-03-02 18:53:17.981041 7fe3640c4ec0 0 set rocksdb option compression = kNoCompression 2019-03-02 18:53:17.981052 7fe3640c4ec0 0 set rocksdb option write_buffer_size = 33554432 2019-03-02 18:53:17.981077 7fe3640c4ec0 0 set rocksdb option compression = kNoCompression 2019-03-02 18:53:17.981084 7fe3640c4ec0 0 set rocksdb option write_buffer_size = 33554432 2019-03-02 18:53:17.981256 7fe3640c4ec0 4 rocksdb: RocksDB version: 5.4.0 2019-03-02 18:53:17.981267 7fe3640c4ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2019-03-02 18:53:17.981269 7fe3640c4ec0 4 rocksdb: Compile date Aug 6 2018 2019-03-02 18:53:17.981279 7fe3640c4ec0 4 rocksdb: DB SUMMARY 2019-03-02 18:53:17.981331 7fe3640c4ec0 4 rocksdb: CURRENT file: CURRENT 2019-03-02 18:53:17.981336 7fe3640c4ec0 4 rocksdb: IDENTITY file: IDENTITY 2019-03-02 18:53:17.981344 7fe3640c4ec0 4 rocksdb: MANIFEST file: MANIFEST-000259 size: 288 Bytes 2019-03-02 18:53:17.981347 7fe3640c4ec0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-storage01roc/store.db dir, Total Num: 4, files: 000250.sst 000252.sst 000255.sst 000258.sst 2019-03-02 18:53:17.981351 7fe3640c4ec0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-storage01roc/store.db: 000260.log size: 49357 ; 2019-03-02 18:53:17.981354 7fe3640c4ec0 4 rocksdb: Options.error_if_exists: 0 2019-03-02 18:53:17.981355 7fe3640c4ec0 4 rocksdb: Options.create_if_missing: 0 2019-03-02 18:53:17.981356 7fe3640c4ec0 4 rocksdb: Options.paranoid_checks: 1 2019-03-02 18:53:17.981360 7fe3640c4ec0 4 rocksdb: Options.env: 0x560103e0c140 2019-03-02 18:53:17.981361 7fe3640c4ec0 4 rocksdb: Options.info_log: 0x56010524ef80 2019-03-02 18:53:17.981362 7fe3640c4ec0 4 rocksdb: Options.max_open_files: -1 2019-03-02 18:53:17.981363 7fe3640c4ec0 4 rocksdb: Options.max_file_opening_threads: 16 2019-03-02 18:53:17.981364 7fe3640c4ec0 4 rocksdb: Options.use_fsync: 0 2019-03-02 18:53:17.981365 7fe3640c4ec0 4 rocksdb: Options.max_log_file_size: 0 2019-03-02 18:53:17.981366 7fe3640c4ec0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2019-03-02 18:53:17.981368 7fe3640c4ec0 4 rocksdb: Options.log_file_time_to_roll: 0 2019-03-02 18:53:17.981369 7fe3640c4ec0 4 rocksdb: Options.keep_log_file_num: 1000 2019-03-02 18:53:17.981370 7fe3640c4ec0 4 rocksdb: Options.recycle_log_file_num: 0 2019-03-02 18:53:17.981371 7fe3640c4ec0 4 rocksdb: Options.allow_fallocate: 1 2019-03-02 18:53:17.981372 7fe3640c4ec0 4 rocksdb: Options.allow_mmap_reads: 0 2019-03-02 18:53:17.981373 7fe3640c4ec0 4 rocksdb: Options.allow_mmap_writes: 0 2019-03-02 18:53:17.981374 7fe3640c4ec0 4 rocksdb: Options.use_direct_reads: 0 2019-03-02 18:53:17.981375 7fe3640c4ec0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2019-03-02 18:53:17.981377 7fe3640c4ec0 4 rocksdb: Options.create_missing_column_families: 0 2019-03-02 18:53:17.981378 7fe3640c4ec0 4 rocksdb: Options.db_log_dir: 2019-03-02 18:53:17.981379 7fe3640c4ec0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-storage01roc/store.db 2019-03-02 18:53:17.981380 7fe3640c4ec0 4 rocksdb: Options.table_cache_numshardbits: 6 2019-03-02 18:53:17.981380 7fe3640c4ec0 4 rocksdb: Options.max_subcompactions: 1 2019-03-02 18:53:17.981381 7fe3640c4ec0 4 rocksdb: Options.max_background_flushes: 1 2019-03-02 18:53:17.981381 7fe3640c4ec0 4 rocksdb: Options.WAL_ttl_seconds: 0 2019-03-02 18:53:17.981382 7fe3640c4ec0 4 rocksdb: Options.WAL_size_limit_MB: 0 2019-03-02 18:53:17.981383 7fe3640c4ec0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2019-03-02 18:53:17.981383 7fe3640c4ec0 4 rocksdb: Options.is_fd_close_on_exec: 1 2019-03-02 18:53:17.981384 7fe3640c4ec0 4 rocksdb: Options.advise_random_on_open: 1 2019-03-02 18:53:17.981385 7fe3640c4ec0 4 rocksdb: Options.db_write_buffer_size: 0 2019-03-02 18:53:17.981385 7fe3640c4ec0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2019-03-02 18:53:17.981386 7fe3640c4ec0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2019-03-02 18:53:17.981386 7fe3640c4ec0 4 rocksdb: Options.compaction_readahead_size: 0 2019-03-02 18:53:17.981387 7fe3640c4ec0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2019-03-02 18:53:17.981388 7fe3640c4ec0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2019-03-02 18:53:17.981388 7fe3640c4ec0 4 rocksdb: Options.use_adaptive_mutex: 0 2019-03-02 18:53:17.981389 7fe3640c4ec0 4 rocksdb: Options.rate_limiter: (nil) 2019-03-02 18:53:17.981390 7fe3640c4ec0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2019-03-02 18:53:17.981391 7fe3640c4ec0 4 rocksdb: Options.bytes_per_sync: 0 2019-03-02 18:53:17.981391 7fe3640c4ec0 4 rocksdb: Options.wal_bytes_per_sync: 0 2019-03-02 18:53:17.981392 7fe3640c4ec0 4 rocksdb: Options.wal_recovery_mode: 2 2019-03-02 18:53:17.981392 7fe3640c4ec0 4 rocksdb: Options.enable_thread_tracking: 0 2019-03-02 18:53:17.981393 7fe3640c4ec0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2019-03-02 18:53:17.981394 7fe3640c4ec0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2019-03-02 18:53:17.981394 7fe3640c4ec0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2019-03-02 18:53:17.981395 7fe3640c4ec0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2019-03-02 18:53:17.981403 7fe3640c4ec0 4 rocksdb: Options.row_cache: None 2019-03-02 18:53:17.981404 7fe3640c4ec0 4 rocksdb: Options.wal_filter: None 2019-03-02 18:53:17.981405 7fe3640c4ec0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2019-03-02 18:53:17.981405 7fe3640c4ec0 4 rocksdb: Options.base_background_compactions: 1 2019-03-02 18:53:17.981406 7fe3640c4ec0 4 rocksdb: Options.max_background_compactions: 1 2019-03-02 18:53:17.981406 7fe3640c4ec0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2019-03-02 18:53:17.981407 7fe3640c4ec0 4 rocksdb: Options.delayed_write_rate : 16777216 2019-03-02 18:53:17.981408 7fe3640c4ec0 4 rocksdb: Options.max_total_wal_size: 0 2019-03-02 18:53:17.981408 7fe3640c4ec0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2019-03-02 18:53:17.981409 7fe3640c4ec0 4 rocksdb: Options.stats_dump_period_sec: 600 2019-03-02 18:53:17.981410 7fe3640c4ec0 4 rocksdb: Compression algorithms supported: 2019-03-02 18:53:17.981412 7fe3640c4ec0 4 rocksdb: Snappy supported: 0 2019-03-02 18:53:17.981413 7fe3640c4ec0 4 rocksdb: Zlib supported: 0 2019-03-02 18:53:17.981413 7fe3640c4ec0 4 rocksdb: Bzip supported: 0 2019-03-02 18:53:17.981414 7fe3640c4ec0 4 rocksdb: LZ4 supported: 0 2019-03-02 18:53:17.981415 7fe3640c4ec0 4 rocksdb: ZSTD supported: 0 2019-03-02 18:53:17.981415 7fe3640c4ec0 4 rocksdb: Fast CRC32 supported: 1 2019-03-02 18:53:17.981735 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000259 2019-03-02 18:53:17.981852 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2019-03-02 18:53:17.981858 7fe3640c4ec0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2019-03-02 18:53:17.981865 7fe3640c4ec0 4 rocksdb: Options.merge_operator: 2019-03-02 18:53:17.981866 7fe3640c4ec0 4 rocksdb: Options.compaction_filter: None 2019-03-02 18:53:17.981866 7fe3640c4ec0 4 rocksdb: Options.compaction_filter_factory: None 2019-03-02 18:53:17.981867 7fe3640c4ec0 4 rocksdb: Options.memtable_factory: SkipListFactory 2019-03-02 18:53:17.981868 7fe3640c4ec0 4 rocksdb: Options.table_factory: BlockBasedTable 2019-03-02 18:53:17.981885 7fe3640c4ec0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560104faa168) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x560105067278 block_cache_name: LRUCache block_cache_options: capacity : 134217728 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 2019-03-02 18:53:17.981894 7fe3640c4ec0 4 rocksdb: Options.write_buffer_size: 33554432 2019-03-02 18:53:17.981895 7fe3640c4ec0 4 rocksdb: Options.max_write_buffer_number: 2 2019-03-02 18:53:17.981896 7fe3640c4ec0 4 rocksdb: Options.compression: NoCompression 2019-03-02 18:53:17.981897 7fe3640c4ec0 4 rocksdb: Options.bottommost_compression: Disabled 2019-03-02 18:53:17.981898 7fe3640c4ec0 4 rocksdb: Options.prefix_extractor: nullptr 2019-03-02 18:53:17.981899 7fe3640c4ec0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2019-03-02 18:53:17.981899 7fe3640c4ec0 4 rocksdb: Options.num_levels: 7 2019-03-02 18:53:17.981900 7fe3640c4ec0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2019-03-02 18:53:17.981901 7fe3640c4ec0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2019-03-02 18:53:17.981901 7fe3640c4ec0 4 rocksdb: Options.compression_opts.window_bits: -14 2019-03-02 18:53:17.981902 7fe3640c4ec0 4 rocksdb: Options.compression_opts.level: -1 2019-03-02 18:53:17.981903 7fe3640c4ec0 4 rocksdb: Options.compression_opts.strategy: 0 2019-03-02 18:53:17.981903 7fe3640c4ec0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2019-03-02 18:53:17.981904 7fe3640c4ec0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2019-03-02 18:53:17.981905 7fe3640c4ec0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2019-03-02 18:53:17.981905 7fe3640c4ec0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2019-03-02 18:53:17.981906 7fe3640c4ec0 4 rocksdb: Options.target_file_size_base: 67108864 2019-03-02 18:53:17.981907 7fe3640c4ec0 4 rocksdb: Options.target_file_size_multiplier: 1 2019-03-02 18:53:17.981907 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2019-03-02 18:53:17.981908 7fe3640c4ec0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2019-03-02 18:53:17.981909 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2019-03-02 18:53:17.981911 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2019-03-02 18:53:17.981915 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2019-03-02 18:53:17.981916 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2019-03-02 18:53:17.981917 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2019-03-02 18:53:17.981921 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2019-03-02 18:53:17.981922 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2019-03-02 18:53:17.981922 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2019-03-02 18:53:17.981923 7fe3640c4ec0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2019-03-02 18:53:17.981923 7fe3640c4ec0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2019-03-02 18:53:17.981924 7fe3640c4ec0 4 rocksdb: Options.arena_block_size: 4194304 2019-03-02 18:53:17.981925 7fe3640c4ec0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2019-03-02 18:53:17.981926 7fe3640c4ec0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2019-03-02 18:53:17.981930 7fe3640c4ec0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2019-03-02 18:53:17.981931 7fe3640c4ec0 4 rocksdb: Options.disable_auto_compactions: 0 2019-03-02 18:53:17.981932 7fe3640c4ec0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2019-03-02 18:53:17.981933 7fe3640c4ec0 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2019-03-02 18:53:17.981934 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2019-03-02 18:53:17.981934 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2019-03-02 18:53:17.981935 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2019-03-02 18:53:17.981936 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2019-03-02 18:53:17.981939 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2019-03-02 18:53:17.981940 7fe3640c4ec0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2019-03-02 18:53:17.981941 7fe3640c4ec0 4 rocksdb: Options.table_properties_collectors: 2019-03-02 18:53:17.981942 7fe3640c4ec0 4 rocksdb: Options.inplace_update_support: 0 2019-03-02 18:53:17.981942 7fe3640c4ec0 4 rocksdb: Options.inplace_update_num_locks: 10000 2019-03-02 18:53:17.981943 7fe3640c4ec0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2019-03-02 18:53:17.981944 7fe3640c4ec0 4 rocksdb: Options.memtable_huge_page_size: 0 2019-03-02 18:53:17.981945 7fe3640c4ec0 4 rocksdb: Options.bloom_locality: 0 2019-03-02 18:53:17.981948 7fe3640c4ec0 4 rocksdb: Options.max_successive_merges: 0 2019-03-02 18:53:17.981949 7fe3640c4ec0 4 rocksdb: Options.optimize_filters_for_hits: 0 2019-03-02 18:53:17.981950 7fe3640c4ec0 4 rocksdb: Options.paranoid_file_checks: 0 2019-03-02 18:53:17.981950 7fe3640c4ec0 4 rocksdb: Options.force_consistency_checks: 0 2019-03-02 18:53:17.981951 7fe3640c4ec0 4 rocksdb: Options.report_bg_io_stats: 0 2019-03-02 18:53:17.982559 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-storage01roc/store.db/MANIFEST-000259 succeeded,manifest_file_number is 259, next_file_number is 261, last_sequence is 15983, log_number is 0,prev_log_number is 0,max_column_family is 0 2019-03-02 18:53:17.982567 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 258 2019-03-02 18:53:17.982631 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997982622, "job": 1, "event": "recovery_started", "log_files": [260]} 2019-03-02 18:53:17.982637 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:482] Recovering log #260 mode 2 2019-03-02 18:53:17.983732 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997983708, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 261, "file_size": 50188, "table_properties": {"data_size": 48886, "index_size": 209, "filter_size": 176, "raw_key_size": 492, "raw_average_key_size": 24, "raw_value_size": 48300, "raw_average_value_size": 2415, "num_data_blocks": 7, "num_entries": 20, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019-03-02 18:53:17.983765 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2395] Creating manifest 262 2019-03-02 18:53:17.984320 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997984316, "job": 1, "event": "recovery_finished"} 2019-03-02 18:53:17.985556 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x560105348000 2019-03-02 18:53:17.985720 7fe355efa700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- 2019-03-02 18:53:17.985728 7fe355efa700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl.cc:448] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 428.46 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 L1 1/0 26.55 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 5/0 26.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 12.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 12.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 428.46 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 L1 1/0 26.55 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 5/0 26.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 12.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2019-03-02 18:53:17.985835 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 4@0 + 1@1 files to L1, score 1.00 2019-03-02 18:53:17.985852 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 0, inputs: [261(49KB) 258(270KB) 255(56KB) 252(53KB)], [250(26MB)] 2019-03-02 18:53:17.985879 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997985856, "job": 3, "event": "compaction_started", "files_L0": [261, 258, 255, 252], "files_L1": [250], "score": 1, "input_data_size": 28275168} 2019-03-02 18:53:17.986482 7fe3576fd700 2 Event(0x560105253480 nevent=5000 time_id=1).set_owner idx=1 owner=140614401251072 2019-03-02 18:53:17.986472 7fe356efc700 2 Event(0x560105258080 nevent=5000 time_id=1).set_owner idx=0 owner=140614392858368 2019-03-02 18:53:17.986516 7fe35d709700 2 Event(0x560105253680 nevent=5000 time_id=1).set_owner idx=2 owner=140614501963520 2019-03-02 18:53:17.986567 7fe3640c4ec0 0 starting mon.storage01roc rank 3 at public addr 192.168.253.203:6789/0 at bind addr 192.168.253.203:6789/0 mon_data /var/lib/ceph/mon/ceph-storage01roc fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea 2019-03-02 18:53:17.986768 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 learned_addr learned my addr 192.168.253.203:6789/0 2019-03-02 18:53:17.986776 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _finish_bind bind my_inst.addr is 192.168.253.203:6789/0 2019-03-02 18:53:17.986792 7fe3640c4ec0 0 starting mon.storage01roc rank 3 at 192.168.253.203:6789/0 mon_data /var/lib/ceph/mon/ceph-storage01roc fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea 2019-03-02 18:53:17.987367 7fe3640c4ec0 1 mon.storage01roc@-1(probing) e14 preinit fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea 2019-03-02 18:53:17.987560 7fe3640c4ec0 1 mon.storage01roc@-1(probing).mds e0 Unable to load 'last_metadata' 2019-03-02 18:53:17.987643 7fe3640c4ec0 1 mon.storage01roc@-1(probing).pg v0 on_upgrade discarding in-core PGMap 2019-03-02 18:53:17.987748 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mds e1 new map 2019-03-02 18:53:17.987752 7fe3640c4ec0 0 mon.storage01roc@-1(probing).mds e1 print_map e1 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2} legacy client fscid: -1 No filesystems configured 2019-03-02 18:53:17.988200 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires 2019-03-02 18:53:17.988209 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires 2019-03-02 18:53:17.988213 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 1009089991638532096, adjusting msgr requires 2019-03-02 18:53:17.988216 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires 2019-03-02 18:53:17.990514 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e0 loading version 519 2019-03-02 18:53:17.990564 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e519 active server: 192.168.253.163:6820/2891295(9267543) 2019-03-02 18:53:17.990592 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e519 mkfs or daemon transitioned to available, loading commands 2019-03-02 18:53:17.990711 7fe3640c4ec0 1 -- - --> 192.168.253.163:6820/2891295 -- mgropen(unknown.storage01roc) v2 -- 0x560104faed80 con 0 2019-03-02 18:53:17.991047 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 start start 2019-03-02 18:53:17.991056 7fe3640c4ec0 1 -- - start start 2019-03-02 18:53:17.991059 7fe3640c4ec0 2 mon.storage01roc@-1(probing) e14 init 2019-03-02 18:53:17.991175 7fe3640c4ec0 1 Processor -- start 2019-03-02 18:53:17.991261 7fe3640c4ec0 1 Processor -- start 2019-03-02 18:53:17.991316 7fe3640c4ec0 0 mon.storage01roc@-1(probing) e14 my rank is now 3 (was -1) 2019-03-02 18:53:17.991322 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 shutdown_connections 2019-03-02 18:53:17.991345 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6780 2019-03-02 18:53:17.991363 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6780 con 0 2019-03-02 18:53:17.991371 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6a00 2019-03-02 18:53:17.991389 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6a00 con 0 2019-03-02 18:53:17.991394 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6c80 2019-03-02 18:53:17.991403 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6c80 con 0 2019-03-02 18:53:17.991407 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6f00 2019-03-02 18:53:17.991418 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6f00 con 0 2019-03-02 18:53:17.991422 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a7180 2019-03-02 18:53:17.991429 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a7180 con 0 2019-03-02 18:53:17.991433 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a7400 2019-03-02 18:53:17.991458 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a7400 con 0 2019-03-02 18:53:17.991794 7fe35d709700 1 -- 192.168.253.203:0/3016783 learned_addr learned my addr 192.168.253.203:0/3016783 2019-03-02 18:53:17.991868 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2577 chose new 2578 2019-03-02 18:53:17.991866 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:17.991889 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:17.991918 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:17.991931 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:17.991939 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2576 chose new 2579 2019-03-02 18:53:17.991962 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:17.991970 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:17.992394 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2577 chose new 2580 2019-03-02 18:53:17.992455 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2576 chose new 2581 2019-03-02 18:53:17.992473 7fe35d709700 2 -- 192.168.253.203:0/3016783 >> 192.168.253.163:6820/2891295 conn(0x56010566f800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 2019-03-02 18:53:17.992706 7fe3576fd700 0 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0).process missed message? skipped from seq 0 to 1798943791 2019-03-02 18:53:17.992727 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943791 0x5601058a7180 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage03roc paxos( fc 13804454 lc 13804982 )) v6 2019-03-02 18:53:17.992761 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943791 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage03roc paxos( fc 13804454 lc 13804982 )) v6 ==== 1014+0+0 (1244369052 0 0) 0x5601058a7180 con 0x560105678800 2019-03-02 18:53:17.992781 7fe356efc700 0 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0).process missed message? skipped from seq 0 to 1756793702 2019-03-02 18:53:17.992795 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection 2019-03-02 18:53:17.992795 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793702 0x5601058a6f00 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage02roc paxos( fc 13804454 lc 13804982 )) v6 2019-03-02 18:53:17.992837 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793702 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage02roc paxos( fc 13804454 lc 13804982 )) v6 ==== 1014+0+0 (4205518415 0 0) 0x5601058a6f00 con 0x560105677000 2019-03-02 18:53:17.992848 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection 2019-03-02 18:53:17.993079 7fe35d709700 5 -- 192.168.253.203:0/3016783 >> 192.168.253.163:6820/2891295 conn(0x56010566f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=8843 cs=1 l=1). rx mgr.9267543 seq 1 0x560105364f00 mgrconfigure(period=5, threshold=5) v2 2019-03-02 18:53:17.993106 7fe3586ff700 1 -- 192.168.253.203:0/3016783 <== mgr.9267543 192.168.253.163:6820/2891295 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x560105364f00 con 0x56010566f800 2019-03-02 18:53:17.993210 7fe3586ff700 1 -- 192.168.253.203:0/3016783 --> 192.168.253.163:6820/2891295 -- mgrreport(unknown.storage01roc +53-0 packed 798) v5 -- 0x5601056b0000 con 0 2019-03-02 18:53:17.993480 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 3480 chose new 3481 2019-03-02 18:53:17.994496 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 3480 chose new 3482 2019-03-02 18:53:17.995791 7fe35d709700 0 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0).process missed message? skipped from seq 0 to 1673216108 2019-03-02 18:53:17.995803 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216108 0x5601058a7400 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon paxos( fc 13804454 lc 13804982 )) v6 2019-03-02 18:53:17.995816 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216108 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon paxos( fc 13804454 lc 13804982 )) v6 ==== 1009+0+0 (1769800333 0 0) 0x5601058a7400 con 0x56010567e000 2019-03-02 18:53:17.995827 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection 2019-03-02 18:53:17.995856 7fe359f02700 0 log_channel(cluster) log [INF] : mon.storage01roc calling monitor election 2019-03-02 18:53:17.995862 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 1 at 2019-03-02 18:53:17.995859) v1 -- 0x5601054f0d80 con 0 2019-03-02 18:53:17.995879 7fe359f02700 5 mon.storage01roc@3(electing).elector(0) start -- can i be leader? 2019-03-02 18:53:17.995896 7fe359f02700 1 mon.storage01roc@3(electing).elector(58452) init, last seen epoch 58452 2019-03-02 18:53:17.996385 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500a300 2019-03-02 18:53:17.996394 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500a300 con 0 2019-03-02 18:53:17.996411 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500a900 2019-03-02 18:53:17.996415 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500a900 con 0 2019-03-02 18:53:17.996433 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500ac00 2019-03-02 18:53:17.996439 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500ac00 con 0 2019-03-02 18:53:17.996453 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500af00 2019-03-02 18:53:17.996458 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500af00 con 0 2019-03-02 18:53:17.996478 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500b200 2019-03-02 18:53:17.996483 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500b200 con 0 2019-03-02 18:53:17.996495 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500b500 2019-03-02 18:53:17.996500 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500b500 con 0 2019-03-02 18:53:17.996514 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 1 at 2019-03-02 18:53:17.995859) v1 ==== 0+0+0 (0 0 0) 0x5601054f0d80 con 0x5601053ea000 2019-03-02 18:53:17.996526 7fe359f02700 5 mon.storage01roc@3(electing) e14 _ms_dispatch setting monitor caps on this connection 2019-03-02 18:53:18.046452 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #266: 3947 keys, 28258990 bytes 2019-03-02 18:53:18.046489 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998046470, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 266, "file_size": 28258990, "table_properties": {"data_size": 28101759, "index_size": 79039, "filter_size": 77265, "raw_key_size": 83782, "raw_average_key_size": 21, "raw_value_size": 27989473, "raw_average_value_size": 7091, "num_data_blocks": 2186, "num_entries": 3947, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019-03-02 18:53:18.046530 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 4@0 + 1@1 files to L1 => 28258990 bytes 2019-03-02 18:53:18.047088 7fe355efa700 4 rocksdb: (Original Log Time 2019/03/02-18:53:18.046986) [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.11, MB/sec: 466.3 rd, 466.1 wr, level 1, files in(4, 1) out(1) MB in(0.4, 26.5) out(26.9), read-write-amplify(128.9) write-amplify(64.4) OK, records in: 3 2019-03-02 18:53:18.047100 7fe355efa700 4 rocksdb: (Original Log Time 2019/03/02-18:53:18.047014) EVENT_LOG_v1 {"time_micros": 1551523998047001, "job": 3, "event": "compaction_finished", "compaction_time_micros": 60635, "output_level": 1, "num_output_files": 1, "total_output_size": 28258990, "num_input_records": 3986, "num_output_records": 3947, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} 2019-03-02 18:53:18.047212 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047206, "job": 3, "event": "table_file_deletion", "file_number": 261} 2019-03-02 18:53:18.047375 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047371, "job": 3, "event": "table_file_deletion", "file_number": 258} 2019-03-02 18:53:18.047456 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047452, "job": 3, "event": "table_file_deletion", "file_number": 255} 2019-03-02 18:53:18.047533 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047529, "job": 3, "event": "table_file_deletion", "file_number": 252} 2019-03-02 18:53:18.055024 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998055019, "job": 3, "event": "table_file_deletion", "file_number": 250} 2019-03-02 18:53:18.113759 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=9 - 2019-03-02 18:53:18.113873 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010543b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=10 - 2019-03-02 18:53:18.113931 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2902 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.114022 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2902 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.114120 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=11 - 2019-03-02 18:53:18.114141 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2902 cs=1 l=1). rx client.8113030 seq 1 0x5601054f06c0 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.114174 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113030 192.168.253.32:0/3724611330 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2331791968 0 0) 0x5601054f06c0 con 0x56010549f000 2019-03-02 18:53:18.114206 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.114261 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13218 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.114326 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13218 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.114375 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=12 - 2019-03-02 18:53:18.114384 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13218 cs=1 l=1). rx client.8831771 seq 1 0x5601054f0480 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.114405 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8831771 192.168.253.43:0/103156267 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3029943862 0 0) 0x5601054f0480 con 0x56010549d800 2019-03-02 18:53:18.114421 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.114456 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279117 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.114924 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279117 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.114961 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279117 cs=1 l=1). rx osd.40 seq 1 0x560104faed80 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.114988 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356684 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.114983 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3408158408 0 0) 0x560104faed80 con 0x56010543b800 2019-03-02 18:53:18.115016 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.115459 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b0000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=13 - 2019-03-02 18:53:18.115512 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356684 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.115540 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356684 cs=1 l=1). rx osd.9 seq 1 0x5601054f1200 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:18.115564 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (32594381 0 0) 0x5601054f1200 con 0x56010549c000 2019-03-02 18:53:18.115584 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:18.115957 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b1800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=28 - 2019-03-02 18:53:18.115975 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b3000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - 2019-03-02 18:53:18.115976 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b4800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=30 - 2019-03-02 18:53:18.116078 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2872 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.116508 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35503 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.116523 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7242 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.116528 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18718 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.116613 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2872 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.116741 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2872 cs=1 l=1). rx client.8775011 seq 1 0x5601054f0900 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.116768 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775011 192.168.253.39:0/1520712163 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2838151096 0 0) 0x5601054f0900 con 0x5601054b0000 2019-03-02 18:53:18.116785 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.117020 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18718 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.117080 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18718 cs=1 l=1). rx client.8833367 seq 1 0x5601054f1680 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.117088 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7242 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.117088 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35503 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.117096 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8833367 192.168.253.50:0/3448699030 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (480927092 0 0) 0x5601054f1680 con 0x5601054b4800 2019-03-02 18:53:18.117108 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.117167 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35503 cs=1 l=1). rx client.8707891 seq 1 0x5601054f18c0 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.117176 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7242 cs=1 l=1). rx client.8707856 seq 1 0x5601054f1d40 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.117185 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8707891 192.168.253.31:0/1087877254 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1012484761 0 0) 0x5601054f18c0 con 0x5601054b1800 2019-03-02 18:53:18.117191 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.117194 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8707856 192.168.253.31:0/1038212884 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1570277103 0 0) 0x5601054f1d40 con 0x5601054b3000 2019-03-02 18:53:18.117199 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:18.177377 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=42 - 2019-03-02 18:53:18.177578 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=14 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.177679 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=14 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.177855 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx client.? seq 1 0x560105492000 auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:18.177877 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.35:0/3362670835 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105492000 con 0x56010548c000 2019-03-02 18:53:18.177894 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:18.191825 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.191829 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.191827 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.191838 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.191840 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.191841 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.341137 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=43 - 2019-03-02 18:53:18.341703 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.342271 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.342333 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6 cs=1 l=1). rx client.? seq 1 0x5601054f1b00 auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:18.342357 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.31:0/2755183883 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x5601054f1b00 con 0x56010548d800 2019-03-02 18:53:18.342379 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:18.429977 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=44 - 2019-03-02 18:53:18.429982 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105490800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=45 - 2019-03-02 18:53:18.430248 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2956 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.430286 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3130 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.430372 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2956 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.430375 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3130 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.430465 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 1 0x5601054926c0 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.430482 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3120407515 0 0) 0x5601054926c0 con 0x560105490800 2019-03-02 18:53:18.430504 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.430511 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 1 0x560105492d80 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.430531 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (1595624629 0 0) 0x560105492d80 con 0x56010548f000 2019-03-02 18:53:18.430538 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:18.591771 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.591772 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.591790 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.591792 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.592768 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:18.592780 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:18.769480 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105638000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=46 - 2019-03-02 18:53:18.769568 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135784 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.769580 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135784 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.769617 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135784 cs=1 l=1). rx osd.35 seq 1 0x560105492fc0 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.769640 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3015307366 0 0) 0x560105492fc0 con 0x560105638000 2019-03-02 18:53:18.769662 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.769663 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105639800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=47 - 2019-03-02 18:53:18.769782 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=86317 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.769829 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=86317 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.769864 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=86317 cs=1 l=1). rx osd.18 seq 1 0x560105492240 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.769878 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.18 192.168.253.205:6816/641907 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3382322512 0 0) 0x560105492240 con 0x560105639800 2019-03-02 18:53:18.769885 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.770288 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=48 - 2019-03-02 18:53:18.770432 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=49 - 2019-03-02 18:53:18.770832 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=465203 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.771033 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3774 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.771300 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=465203 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.771340 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 1 0x560105492900 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.771356 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3123283281 0 0) 0x560105492900 con 0x56010563b000 2019-03-02 18:53:18.771365 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:18.771496 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3774 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.771534 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3774 cs=1 l=1). rx client.9267543 seq 1 0x560105493200 auth(proto 0 37 bytes epoch 14) v1 2019-03-02 18:53:18.771549 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 1 ==== auth(proto 0 37 bytes epoch 14) v1 ==== 67+0+0 (3305926266 0 0) 0x560105493200 con 0x56010563c800 2019-03-02 18:53:18.771557 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 37 bytes epoch 14) v1 2019-03-02 18:53:18.785093 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545a000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=50 - 2019-03-02 18:53:18.785186 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=95151 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:18.785233 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=95151 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:18.785268 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 1 0x560105492480 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:18.785294 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2633358351 0 0) 0x560105492480 con 0x56010545a000 2019-03-02 18:53:18.785310 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:19.132954 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=51 - 2019-03-02 18:53:19.132999 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545d000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=52 - 2019-03-02 18:53:19.133056 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=96529 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133097 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=96529 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.133130 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=96529 cs=1 l=1). rx osd.12 seq 1 0x560105492b40 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.133154 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.12 192.168.253.205:6814/641212 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (2274412844 0 0) 0x560105492b40 con 0x56010545b800 2019-03-02 18:53:19.133172 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.133305 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545e800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=53 - 2019-03-02 18:53:19.133498 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f4000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=54 - 2019-03-02 18:53:19.133521 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=353413 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133639 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2943 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133696 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2943 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.133747 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2943 cs=1 l=1). rx client.8775228 seq 1 0x560105493440 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.133760 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f5800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=55 - 2019-03-02 18:53:19.133767 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f7000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=56 - 2019-03-02 18:53:19.133764 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775228 192.168.253.32:0/360631102 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (1894617780 0 0) 0x560105493440 con 0x5601056f4000 2019-03-02 18:53:19.133800 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f8800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=57 - 2019-03-02 18:53:19.133803 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.133835 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=504598 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133853 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105758000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=58 - 2019-03-02 18:53:19.133927 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=49224 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133966 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7222 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.133970 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=353413 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.133999 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=353413 cs=1 l=1). rx osd.0 seq 1 0x560105493d40 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:19.134025 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=49224 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.134022 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.0 192.168.253.164:6806/342760 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2286119786 0 0) 0x560105493d40 con 0x56010545d000 2019-03-02 18:53:19.134038 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:19.134047 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7222 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.134085 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=49224 cs=1 l=1). rx client.8947140 seq 1 0x560105434000 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:19.134101 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8947140 192.168.253.46:0/836723901 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3760714466 0 0) 0x560105434000 con 0x5601056f5800 2019-03-02 18:53:19.134107 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 1 0x560105434b40 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.134117 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:19.134120 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3816987468 0 0) 0x560105434b40 con 0x560105758000 2019-03-02 18:53:19.134127 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.134254 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105759800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=59 - 2019-03-02 18:53:19.134305 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=504598 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.134315 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=411209 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.134342 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=504598 cs=1 l=1). rx osd.8 seq 1 0x560105434d80 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:19.134356 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.8 192.168.253.163:6812/345669 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (1449422260 0 0) 0x560105434d80 con 0x56010545e800 2019-03-02 18:53:19.134373 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:19.134397 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2894 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.134429 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6880 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.134495 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6880 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.134571 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6880 cs=1 l=1). rx client.8945978 seq 1 0x560105493680 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:19.134586 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8945978 192.168.253.46:0/2587405440 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (434972468 0 0) 0x560105493680 con 0x560105759800 2019-03-02 18:53:19.134600 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:19.134803 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=411209 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.134840 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=411209 cs=1 l=1). rx osd.45 seq 1 0x560105434240 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.134854 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.45 192.168.253.165:6812/710509 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (2375892178 0 0) 0x560105434240 con 0x5601056f8800 2019-03-02 18:53:19.134862 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.134919 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2894 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.135009 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2894 cs=1 l=1). rx client.8112890 seq 1 0x5601054938c0 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.135023 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8112890 192.168.253.39:0/282811283 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (435497577 0 0) 0x5601054938c0 con 0x5601056f7000 2019-03-02 18:53:19.135028 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:19.292967 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010575b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=60 - 2019-03-02 18:53:19.293498 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=189842 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.293957 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=189842 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.293985 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 1 0x560105434480 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.293999 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3917320626 0 0) 0x560105434480 con 0x56010575b000 2019-03-02 18:53:19.294016 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.391772 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:19.391785 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:19.391785 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:19.391799 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:19.392773 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:19.392785 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:19.782889 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).read_bulk peer close file descriptor 51 2019-03-02 18:53:19.782904 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).read_until read failed 2019-03-02 18:53:19.782916 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).process read tag failed 2019-03-02 18:53:19.782923 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:19.782928 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1)._stop 2019-03-02 18:53:19.861758 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).read_bulk peer close file descriptor 49 2019-03-02 18:53:19.861771 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).read_until read failed 2019-03-02 18:53:19.861780 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).process read tag failed 2019-03-02 18:53:19.861787 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:19.861792 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1)._stop 2019-03-02 18:53:19.862298 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010575c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=49 - 2019-03-02 18:53:19.862849 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3778 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.863299 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3778 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.863331 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 1 0x5601054346c0 auth(proto 0 37 bytes epoch 14) v1 2019-03-02 18:53:19.863352 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 1 ==== auth(proto 0 37 bytes epoch 14) v1 ==== 67+0+0 (3305926266 0 0) 0x5601054346c0 con 0x56010575c800 2019-03-02 18:53:19.863374 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 37 bytes epoch 14) v1 2019-03-02 18:53:19.891302 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).read_bulk peer close file descriptor 10 2019-03-02 18:53:19.891314 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).read_until read failed 2019-03-02 18:53:19.891319 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).process read tag failed 2019-03-02 18:53:19.891322 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:19.891325 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1)._stop 2019-03-02 18:53:19.891847 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105772000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=10 - 2019-03-02 18:53:19.892399 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279118 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:19.892857 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279118 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:19.892885 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 1 0x560105434fc0 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:19.892907 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3408158408 0 0) 0x560105434fc0 con 0x560105772000 2019-03-02 18:53:19.892926 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:20.187672 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105773800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=51 - 2019-03-02 18:53:20.187740 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=89395 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:20.187749 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=89395 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:20.187790 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 1 0x560105434900 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:20.187808 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2020154301 0 0) 0x560105434900 con 0x560105773800 2019-03-02 18:53:20.187826 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:20.527395 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105775000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=61 - 2019-03-02 18:53:20.527559 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11207 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:20.527629 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11207 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:20.527710 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 1 0x560105435200 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.527730 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (2078396002 0 0) 0x560105435200 con 0x560105775000 2019-03-02 18:53:20.527747 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.738669 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).read_bulk peer close file descriptor 43 2019-03-02 18:53:20.738683 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).read_until read failed 2019-03-02 18:53:20.738694 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).process read tag failed 2019-03-02 18:53:20.738701 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:20.738707 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1)._stop 2019-03-02 18:53:20.753597 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105776800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=43 - 2019-03-02 18:53:20.753639 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105788000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=62 - 2019-03-02 18:53:20.754182 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12571 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:20.754213 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=4540 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:20.754755 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12571 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:20.754777 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=4540 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:20.754825 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 1 0x560105493b00 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.754839 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (4053673190 0 0) 0x560105493b00 con 0x560105776800 2019-03-02 18:53:20.754859 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.754860 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 1 0x560105435440 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.754872 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1470431156 0 0) 0x560105435440 con 0x560105788000 2019-03-02 18:53:20.754881 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.811243 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105789800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=63 - 2019-03-02 18:53:20.811396 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6463 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:20.811494 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6463 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:20.811612 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 1 0x560105794000 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.811633 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3722122544 0 0) 0x560105794000 con 0x560105789800 2019-03-02 18:53:20.811650 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:20.991777 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:20.991778 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:20.991792 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:20.991794 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:20.992771 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:20.992782 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:21.177128 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).read_bulk peer close file descriptor 42 2019-03-02 18:53:21.177141 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).read_until read failed 2019-03-02 18:53:21.177145 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).process read tag failed 2019-03-02 18:53:21.177149 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:21.177152 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1)._stop 2019-03-02 18:53:21.192635 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010578b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=42 - 2019-03-02 18:53:21.193203 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=54339 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:21.193861 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=54339 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:21.194000 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 1 0x560105794d80 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:21.194022 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.7893124 192.168.253.44:0/219447902 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3817785262 0 0) 0x560105794d80 con 0x56010578b000 2019-03-02 18:53:21.194041 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:21.281260 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010578c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=64 - 2019-03-02 18:53:21.281445 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=28 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:21.281607 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=28 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:21.281697 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=28 cs=1 l=1). rx client.? seq 1 0x560105794240 auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:21.281719 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.46:0/2878661900 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105794240 con 0x56010578c800 2019-03-02 18:53:21.281737 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:21.985027 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058f8000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=65 - 2019-03-02 18:53:21.985125 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058f9800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=66 - 2019-03-02 18:53:21.985617 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18715 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:21.985681 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13944 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:21.986105 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18715 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:21.986193 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 1 0x560105794fc0 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:21.986209 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (875330357 0 0) 0x560105794fc0 con 0x5601058f8000 2019-03-02 18:53:21.986228 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:21.986266 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13944 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:21.986337 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 1 0x560105435680 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:21.986354 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (2336192628 0 0) 0x560105435680 con 0x5601058f9800 2019-03-02 18:53:21.986367 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:22.141538 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793703 0x56010500af00 election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58459) v7 2019-03-02 18:53:22.141575 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793703 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58459) v7 ==== 1023+0+0 (831641765 0 0) 0x56010500af00 con 0x560105677000 2019-03-02 18:53:22.141623 7fe359f02700 5 mon.storage01roc@3(electing).elector(58453) handle_propose from mon.4 2019-03-02 18:53:22.141840 7fe359f02700 0 log_channel(cluster) log [INF] : mon.storage01roc calling monitor election 2019-03-02 18:53:22.141849 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 2 at 2019-03-02 18:53:22.141845) v1 -- 0x5601057958c0 con 0 2019-03-02 18:53:22.141865 7fe359f02700 5 mon.storage01roc@3(electing).elector(58459) start -- can i be leader? 2019-03-02 18:53:22.141875 7fe359f02700 1 mon.storage01roc@3(electing).elector(58459) init, last seen epoch 58459, mid-election, bumping 2019-03-02 18:53:22.142435 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x56010500b800 2019-03-02 18:53:22.142447 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x56010500b800 con 0 2019-03-02 18:53:22.142464 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x56010500bb00 2019-03-02 18:53:22.142468 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x56010500bb00 con 0 2019-03-02 18:53:22.142499 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908000 2019-03-02 18:53:22.142511 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908000 con 0 2019-03-02 18:53:22.142535 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908300 2019-03-02 18:53:22.142539 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908300 con 0 2019-03-02 18:53:22.142552 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908600 2019-03-02 18:53:22.142561 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908600 con 0 2019-03-02 18:53:22.142580 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908900 2019-03-02 18:53:22.142588 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908900 con 0 2019-03-02 18:53:22.142613 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 2 at 2019-03-02 18:53:22.141845) v1 ==== 0+0+0 (0 0 0) 0x5601057958c0 con 0x5601053ea000 2019-03-02 18:53:22.143244 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943792 0x560105908600 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 2019-03-02 18:53:22.143270 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943792 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40498+0+0 (3500033299 0 0) 0x560105908600 con 0x560105678800 2019-03-02 18:53:22.143300 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793704 0x560105908300 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 2019-03-02 18:53:22.143323 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.5 2019-03-02 18:53:22.143329 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]) } 2019-03-02 18:53:22.143345 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793704 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40498+0+0 (2531979772 0 0) 0x560105908300 con 0x560105677000 2019-03-02 18:53:22.143368 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.4 2019-03-02 18:53:22.143373 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.4: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]) } 2019-03-02 18:53:22.153619 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216109 0x560105908900 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 2019-03-02 18:53:22.153647 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216109 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40488+0+0 (2345732302 0 0) 0x560105908900 con 0x56010567e000 2019-03-02 18:53:22.153679 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.6 2019-03-02 18:53:22.153686 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.4: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.6: features 2305244844532236283 mon_feature_t([kraken,luminous]) } 2019-03-02 18:53:22.993342 7fe358f00700 1 -- 192.168.253.203:0/3016783 --> 192.168.253.163:6820/2891295 -- mgrreport(unknown.storage01roc +0-0 packed 798) v5 -- 0x5601056b02c0 con 0 2019-03-02 18:53:23.063662 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058fb000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=67 - 2019-03-02 18:53:23.064262 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=5417 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:23.064813 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=5417 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:23.064945 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 1 0x560105794480 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:23.064968 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2725449133 0 0) 0x560105794480 con 0x5601058fb000 2019-03-02 18:53:23.064988 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:23.086826 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058fc800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=68 - 2019-03-02 18:53:23.087017 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11821 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:23.087130 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11821 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:23.087292 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 1 0x560105795200 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:23.087313 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (98846487 0 0) 0x560105795200 con 0x5601058fc800 2019-03-02 18:53:23.087329 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:23.240286 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).read_bulk peer close file descriptor 29 2019-03-02 18:53:23.240299 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).read_until read failed 2019-03-02 18:53:23.240303 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).process read tag failed 2019-03-02 18:53:23.240309 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:23.240312 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1)._stop 2019-03-02 18:53:23.740441 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - 2019-03-02 18:53:23.741038 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:23.741515 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:23.741671 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx client.? seq 1 0x560105795440 auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:23.741689 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.31:0/2755183883 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105795440 con 0x56010563c800 2019-03-02 18:53:23.741706 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:23.818121 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).read_bulk peer close file descriptor 11 2019-03-02 18:53:23.818134 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).read_until read failed 2019-03-02 18:53:23.818137 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).process read tag failed 2019-03-02 18:53:23.818142 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:23.818145 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1)._stop 2019-03-02 18:53:23.858950 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010543b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=11 - 2019-03-02 18:53:23.858973 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010593e000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=69 - 2019-03-02 18:53:23.859494 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6891 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:23.859531 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35760 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:23.860064 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6891 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:23.860116 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35760 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:23.860631 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 1 0x5601054358c0 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:23.860636 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 1 0x560105795680 auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:23.860655 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3214538620 0 0) 0x5601054358c0 con 0x56010543b800 2019-03-02 18:53:23.860675 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:23.860683 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (701291500 0 0) 0x560105795680 con 0x56010593e000 2019-03-02 18:53:23.860690 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 2019-03-02 18:53:24.153794 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).read_bulk peer close file descriptor 52 2019-03-02 18:53:24.153812 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).read_until read failed 2019-03-02 18:53:24.153818 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).process read tag failed 2019-03-02 18:53:24.153824 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:24.153827 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1)._stop 2019-03-02 18:53:24.153837 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start 2019-03-02 18:53:24.153839 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545b800 2019-03-02 18:53:24.153844 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545d000 2019-03-02 18:53:24.153846 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010548c000 2019-03-02 18:53:24.153852 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010549d800 2019-03-02 18:53:24.153856 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b3000 2019-03-02 18:53:24.191773 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:24.191775 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:24.191787 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:24.191791 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:24.192771 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed 2019-03-02 18:53:24.192783 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! 2019-03-02 18:53:24.281656 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).read_bulk peer close file descriptor 64 2019-03-02 18:53:24.281668 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).read_until read failed 2019-03-02 18:53:24.281672 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).process read tag failed 2019-03-02 18:53:24.281686 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:24.281690 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1)._stop 2019-03-02 18:53:24.298590 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).read_bulk peer close file descriptor 13 2019-03-02 18:53:24.298603 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).read_until read failed 2019-03-02 18:53:24.298607 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).process read tag failed 2019-03-02 18:53:24.298610 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:24.298613 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1)._stop 2019-03-02 18:53:24.299896 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).read_bulk peer close file descriptor 56 2019-03-02 18:53:24.299908 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).read_until read failed 2019-03-02 18:53:24.299912 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).process read tag failed 2019-03-02 18:53:24.299916 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:24.299919 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1)._stop 2019-03-02 18:53:24.557504 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=13 - 2019-03-02 18:53:24.557696 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=1 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:24.557765 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=1 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:24.557881 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1 cs=1 l=1). rx client.? seq 1 0x5601057946c0 auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:24.557899 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.52:0/3887599110 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x5601057946c0 con 0x56010548c000 2019-03-02 18:53:24.557916 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 2019-03-02 18:53:24.685869 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).read_bulk peer close file descriptor 53 2019-03-02 18:53:24.685883 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).read_until read failed 2019-03-02 18:53:24.685886 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).process read tag failed 2019-03-02 18:53:24.685890 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:24.685893 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1)._stop 2019-03-02 18:53:25.253219 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=52 - 2019-03-02 18:53:25.253831 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=330841 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.254298 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=330841 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.254349 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 1 0x560105493200 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.254367 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (4213901812 0 0) 0x560105493200 con 0x56010545b800 2019-03-02 18:53:25.254384 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.293984 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).read_bulk peer close file descriptor 30 2019-03-02 18:53:25.293997 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.294001 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.294004 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.294008 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1)._stop 2019-03-02 18:53:25.294017 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start 2019-03-02 18:53:25.294019 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545e800 2019-03-02 18:53:25.294033 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b0000 2019-03-02 18:53:25.294037 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b4800 2019-03-02 18:53:25.294040 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f7000 2019-03-02 18:53:25.294043 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010578c800 2019-03-02 18:53:25.397219 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).read_bulk peer close file descriptor 59 2019-03-02 18:53:25.397232 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.397236 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.397240 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.397243 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1)._stop 2019-03-02 18:53:25.452621 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010593f800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=30 - 2019-03-02 18:53:25.452801 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=68711 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.452839 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=68711 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.452903 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 1 0x560105794900 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:25.452923 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (3612044724 0 0) 0x560105794900 con 0x56010593f800 2019-03-02 18:53:25.452942 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:25.624523 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).read_bulk peer close file descriptor 55 2019-03-02 18:53:25.624536 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.624541 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.624544 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.624547 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1)._stop 2019-03-02 18:53:25.636883 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105941000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=53 - 2019-03-02 18:53:25.637444 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=245753 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.637900 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=245753 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.637927 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 1 0x560105794b40 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.637941 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3528568399 0 0) 0x560105794b40 con 0x560105941000 2019-03-02 18:53:25.637958 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.761886 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).read_bulk peer close file descriptor 12 2019-03-02 18:53:25.761899 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.761903 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.761907 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.761910 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1)._stop 2019-03-02 18:53:25.762407 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105942800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=12 - 2019-03-02 18:53:25.762941 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356689 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.763392 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356689 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.763427 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 1 0x560105958000 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:25.763441 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (32594381 0 0) 0x560105958000 con 0x560105942800 2019-03-02 18:53:25.763458 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:25.775208 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).read_bulk peer close file descriptor 46 2019-03-02 18:53:25.775219 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.775223 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.775226 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.775235 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1)._stop 2019-03-02 18:53:25.775308 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=46 - 2019-03-02 18:53:25.775369 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135785 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.775378 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135785 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.775413 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 1 0x560105958240 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.775431 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3015307366 0 0) 0x560105958240 con 0x56010595c000 2019-03-02 18:53:25.775453 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:25.827202 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).read_bulk peer close file descriptor 9 2019-03-02 18:53:25.827216 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.827219 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.827223 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.827226 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1)._stop 2019-03-02 18:53:25.827256 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).read_bulk peer close file descriptor 54 2019-03-02 18:53:25.827268 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).read_until read failed 2019-03-02 18:53:25.827272 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).process read tag failed 2019-03-02 18:53:25.827283 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:25.827286 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1)._stop 2019-03-02 18:53:25.827414 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=54 - 2019-03-02 18:53:25.827422 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=9 - 2019-03-02 18:53:25.827490 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105960800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=55 - 2019-03-02 18:53:25.827532 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2952 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.827574 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3075 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.827624 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3075 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.827625 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3017 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:25.827648 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2952 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.827666 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 1 0x560105435b00 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:25.827680 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3011287488 0 0) 0x560105435b00 con 0x56010595f000 2019-03-02 18:53:25.827698 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:25.827704 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3017 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:25.827710 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 1 0x560105958480 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:25.827723 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2551472749 0 0) 0x560105958480 con 0x56010595d800 2019-03-02 18:53:25.827732 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:25.827747 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 1 0x560105435d40 auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:25.827767 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (133202792 0 0) 0x560105435d40 con 0x560105960800 2019-03-02 18:53:25.827775 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 2019-03-02 18:53:26.057844 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).read_bulk peer close file descriptor 47 2019-03-02 18:53:26.057856 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).read_until read failed 2019-03-02 18:53:26.057860 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).process read tag failed 2019-03-02 18:53:26.057863 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:26.057866 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1)._stop 2019-03-02 18:53:26.057887 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start 2019-03-02 18:53:26.057897 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010549f000 2019-03-02 18:53:26.057901 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x560105639800 2019-03-02 18:53:26.057904 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f4000 2019-03-02 18:53:26.057906 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f5800 2019-03-02 18:53:26.057908 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x560105759800 2019-03-02 18:53:26.716146 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105972000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=47 - 2019-03-02 18:53:26.716279 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=90617 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:26.716312 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=90617 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:26.716352 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 1 0x560105492b40 auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:26.716375 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2372153616 0 0) 0x560105492b40 con 0x560105972000 2019-03-02 18:53:26.716396 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 2019-03-02 18:53:26.739866 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).read_bulk peer close file descriptor 29 2019-03-02 18:53:26.739879 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).read_until read failed 2019-03-02 18:53:26.739883 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).process read tag failed 2019-03-02 18:53:26.739887 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:26.739890 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1)._stop 2019-03-02 18:53:26.755304 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).read_bulk peer close file descriptor 57 2019-03-02 18:53:26.755316 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).read_until read failed 2019-03-02 18:53:26.755320 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).process read tag failed 2019-03-02 18:53:26.755323 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:26.755326 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1)._stop 2019-03-02 18:53:26.832047 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105973800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - 2019-03-02 18:53:26.832645 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=454471 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:26.833106 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=454471 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:26.833135 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 1 0x560105492000 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:26.833152 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (317843293 0 0) 0x560105492000 con 0x560105973800 2019-03-02 18:53:26.833170 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:26.936709 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105975000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=56 - 2019-03-02 18:53:26.936900 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87095 cs=1 l=1).handle_connect_msg accept write reply msg done 2019-03-02 18:53:26.936931 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87095 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2019-03-02 18:53:26.936962 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87095 cs=1 l=1). rx osd.24 seq 1 0x560105958fc0 auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:26.936977 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.24 192.168.253.205:6818/642558 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3667517188 0 0) 0x560105958fc0 con 0x560105975000 2019-03-02 18:53:26.936994 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 2019-03-02 18:53:26.992061 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).read_bulk peer close file descriptor 28 2019-03-02 18:53:26.992074 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).read_until read failed 2019-03-02 18:53:26.992078 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).process read tag failed 2019-03-02 18:53:26.992081 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:26.992091 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1)._stop 2019-03-02 18:53:27.142660 7fe35c707700 5 mon.storage01roc@3(electing).elector(58461) election timer expired 2019-03-02 18:53:27.142900 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105908f00 2019-03-02 18:53:27.142919 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105908f00 con 0 2019-03-02 18:53:27.142940 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105909200 2019-03-02 18:53:27.142946 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105909200 con 0 2019-03-02 18:53:27.142987 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105909500 2019-03-02 18:53:27.143002 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105909500 con 0 2019-03-02 18:53:27.143051 7fe35c707700 0 log_channel(cluster) log [INF] : mon.storage01roc is new leader, mons storage01roc,storage02roc,storage03roc,cephmon in quorum (ranks 3,4,5,6) 2019-03-02 18:53:27.143066 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 3 at 2019-03-02 18:53:27.143055) v1 -- 0x560105992000 con 0 2019-03-02 18:53:27.143107 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 3 at 2019-03-02 18:53:27.143055) v1 ==== 0+0+0 (0 0 0) 0x560105992000 con 0x5601053ea000 2019-03-02 18:53:27.143292 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x560105909800 2019-03-02 18:53:27.143304 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x560105909800 con 0 2019-03-02 18:53:27.143314 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x560105909b00 2019-03-02 18:53:27.143319 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x560105909b00 con 0 2019-03-02 18:53:27.143325 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x56010598e000 2019-03-02 18:53:27.143329 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x56010598e000 con 0 2019-03-02 18:53:27.143411 7fe35c707700 5 mon.storage01roc@3(leader) e14 apply_quorum_to_compatset_features 2019-03-02 18:53:27.143421 7fe35c707700 5 mon.storage01roc@3(leader) e14 apply_monmap_to_compatset_features 2019-03-02 18:53:27.143486 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105795440 2019-03-02 18:53:27.143494 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105795440 con 0 2019-03-02 18:53:27.143502 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105959d40 2019-03-02 18:53:27.143512 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105959d40 con 0 2019-03-02 18:53:27.143520 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105992480 2019-03-02 18:53:27.143524 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105992480 con 0 2019-03-02 18:53:27.143610 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943793 0x5601058a7180 forward(log(1 entries from seq 249 at 2019-03-02 18:53:00.285700) v1 caps allow * tid 3934 con_features 2305244844532236283) v3 2019-03-02 18:53:27.143638 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943793 ==== forward(log(1 entries from seq 249 at 2019-03-02 18:53:00.285700) v1 caps allow * tid 3934 con_features 2305244844532236283) v3 ==== 451+0+0 (2345767367 0 0) 0x5601058a7180 con 0x560105678800 2019-03-02 18:53:27.143688 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943794 0x56010599c000 forward(log(1 entries from seq 250 at 2019-03-02 18:53:00.286175) v1 caps allow * tid 3935 con_features 2305244844532236283) v3 2019-03-02 18:53:27.143715 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943794 ==== forward(log(1 entries from seq 250 at 2019-03-02 18:53:00.286175) v1 caps allow * tid 3935 con_features 2305244844532236283) v3 ==== 521+0+0 (1805236427 0 0) 0x56010599c000 con 0x560105678800 2019-03-02 18:53:27.144024 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793705 0x560105908f00 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 2019-03-02 18:53:27.144044 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793705 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (1918868918 0 0) 0x560105908f00 con 0x560105677000 2019-03-02 18:53:27.144053 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793706 0x560105795440 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144163 ) v1 2019-03-02 18:53:27.144074 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793706 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144163 ) v1 ==== 36+0+0 (810820296 0 0) 0x560105795440 con 0x560105677000 2019-03-02 18:53:27.144108 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943795 0x560105909b00 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 2019-03-02 18:53:27.144128 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943795 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (3193165242 0 0) 0x560105909b00 con 0x560105678800 2019-03-02 18:53:27.144133 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943796 0x560105959d40 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144374 ) v1 2019-03-02 18:53:27.144149 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943796 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144374 ) v1 ==== 36+0+0 (546103651 0 0) 0x560105959d40 con 0x560105678800 2019-03-02 18:53:27.155849 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216110 0x56010598e000 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 2019-03-02 18:53:27.155869 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216110 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (1592508016 0 0) 0x56010598e000 con 0x56010567e000 2019-03-02 18:53:27.155873 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216111 0x560105992480 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.154851 ) v1 2019-03-02 18:53:27.155884 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908c00 2019-03-02 18:53:27.155889 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908c00 con 0 2019-03-02 18:53:27.155899 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908600 2019-03-02 18:53:27.155904 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908600 con 0 2019-03-02 18:53:27.155911 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908300 2019-03-02 18:53:27.155915 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908300 con 0 2019-03-02 18:53:27.156054 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943797 0x560105908600 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 2019-03-02 18:53:27.156055 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793707 0x560105908c00 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 2019-03-02 18:53:27.156170 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943798 0x56010599c280 forward(log(1 entries from seq 251 at 2019-03-02 18:53:07.139680) v1 caps allow * tid 3940 con_features 2305244844532236283) v3 2019-03-02 18:53:27.156204 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793708 0x5601058a7400 forward(log(1 entries from seq 352 at 2019-03-02 18:53:07.138814) v1 caps allow * tid 5274 con_features 2305244844532236283) v3 2019-03-02 18:53:27.156302 7fe359f02700 0 log_channel(cluster) log [DBG] : monmap e14: 7 mons at {cephmon=192.168.253.242:6789/0,storage01moc=192.168.253.163:6789/0,storage01roc=192.168.253.203:6789/0,storage02moc=192.168.253.164:6789/0,storage02roc=192.168.253.204:6789/0,storage03moc=192.168.253.165:6789/0,storage03roc=192.168.253.205:6789/0} 2019-03-02 18:53:27.156309 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 4 at 2019-03-02 18:53:27.156308) v1 -- 0x560105795440 con 0 2019-03-02 18:53:27.156344 7fe359f02700 0 log_channel(cluster) log [DBG] : fsmap 2019-03-02 18:53:27.156348 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 5 at 2019-03-02 18:53:27.156347) v1 -- 0x560105795b00 con 0 2019-03-02 18:53:27.156418 7fe359f02700 0 log_channel(cluster) log [DBG] : osdmap e79198: 60 total, 60 up, 60 in 2019-03-02 18:53:27.156422 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 6 at 2019-03-02 18:53:27.156421) v1 -- 0x560105795d40 con 0 2019-03-02 18:53:27.156482 7fe359f02700 0 log_channel(cluster) log [DBG] : mgrmap e519: storage01moc(active), standbys: storage02roc, storage03roc, storage03moc, storage01roc, cephmon 2019-03-02 18:53:27.156486 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 7 at 2019-03-02 18:53:27.156485) v1 -- 0x560105992900 con 0 2019-03-02 18:53:27.156679 7fe359f02700 0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 3/7 mons down, quorum storage01roc,storage02roc,storage03roc,cephmon 2019-03-02 18:53:27.156685 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 8 at 2019-03-02 18:53:27.156683) v1 -- 0x560105992d80 con 0 2019-03-02 18:53:27.156717 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216112 0x560105908300 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 2019-03-02 18:53:27.156915 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601053838c0 0x5601054f1d40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.156929 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3361644023 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054f1d40 con 0 2019-03-02 18:53:27.156965 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601053839e0 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.156974 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3589307736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 2019-03-02 18:53:27.157007 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547a480 0x560105492240 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157012 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492240 con 0 2019-03-02 18:53:27.157029 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547a6c0 0x560105492900 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157033 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492900 con 0 2019-03-02 18:53:27.157060 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547b560 0x560105434000 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157065 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/614279813 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434000 con 0 2019-03-02 18:53:27.157095 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547bb00 0x5601054938c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157100 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054938c0 con 0 2019-03-02 18:53:27.157117 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770000 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157123 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 2019-03-02 18:53:27.157151 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770120 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157158 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 2019-03-02 18:53:27.157178 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770240 0x560105434fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157181 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434fc0 con 0 2019-03-02 18:53:27.157196 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770360 0x560105434900 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157199 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/94488736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434900 con 0 2019-03-02 18:53:27.157217 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770480 0x560105435200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157222 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3202181189 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435200 con 0 2019-03-02 18:53:27.157217 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 2 0x5601054346c0 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157247 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057705a0 0x560105493b00 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157250 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3953751394 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493b00 con 0 2019-03-02 18:53:27.157265 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057706c0 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157267 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/1267131636 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 2019-03-02 18:53:27.157294 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057707e0 0x560105794000 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157297 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.44:0/219447902 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794000 con 0 2019-03-02 18:53:27.157317 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105771c20 0x560105794240 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157319 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3482224786 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794240 con 0 2019-03-02 18:53:27.157333 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105771d40 0x560105794fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157337 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3137663203 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794fc0 con 0 2019-03-02 18:53:27.157354 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928000 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157358 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.33:0/140057803 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 2019-03-02 18:53:27.157360 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 2 0x560105435440 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157373 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928120 0x560105794480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157375 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.35:0/3274667345 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794480 con 0 2019-03-02 18:53:27.157383 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 2 0x560105434fc0 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.157391 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928360 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157383 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 2 0x560105493b00 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.157394 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/1733074246 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 2019-03-02 18:53:27.157406 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928480 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157408 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/2175704180 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 2019-03-02 18:53:27.157508 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 2 0x5601054358c0 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.157591 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 2 0x560105795200 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157631 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).read_bulk peer close file descriptor 56 2019-03-02 18:53:27.157647 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).read_until read failed 2019-03-02 18:53:27.157655 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).process read tag failed 2019-03-02 18:53:27.157662 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 2 0x560105435680 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157663 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).fault on lossy channel, failing 2019-03-02 18:53:27.157668 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1)._stop 2019-03-02 18:53:27.157709 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 2 0x560105492900 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.157715 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 2 0x560105434480 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157743 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 2 0x5601054938c0 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.157740 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x560105909b00 2019-03-02 18:53:27.157746 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 2 0x560105493d40 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.157754 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x560105909b00 con 0 2019-03-02 18:53:27.157765 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x56010500af00 2019-03-02 18:53:27.157769 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x56010500af00 con 0 2019-03-02 18:53:27.157778 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x56010598ec00 2019-03-02 18:53:27.157782 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x56010598ec00 con 0 2019-03-02 18:53:27.157810 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059286c0 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157819 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 2019-03-02 18:53:27.157839 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059287e0 0x560105493200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157844 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493200 con 0 2019-03-02 18:53:27.157870 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928900 0x560105794900 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157876 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794900 con 0 2019-03-02 18:53:27.157895 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928a20 0x560105794b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157900 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794b40 con 0 2019-03-02 18:53:27.157923 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a900 0x560105958000 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157928 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958000 con 0 2019-03-02 18:53:27.157945 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595aa20 0x560105958240 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157949 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/1251637173 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958240 con 0 2019-03-02 18:53:27.157966 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ab40 0x560105435b00 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157970 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/683377982 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435b00 con 0 2019-03-02 18:53:27.157986 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ac60 0x560105958480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.157990 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/419113882 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958480 con 0 2019-03-02 18:53:27.158007 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ad80 0x560105435d40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158012 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435d40 con 0 2019-03-02 18:53:27.158011 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 2 0x5601054926c0 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158028 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595aea0 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158033 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 2019-03-02 18:53:27.158038 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 2 0x560105493200 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158061 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 2 0x560105492fc0 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158081 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216111 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.154851 ) v1 ==== 36+0+0 (282032588 0 0) 0x560105992480 con 0x56010567e000 2019-03-02 18:53:27.158102 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105958fc0 2019-03-02 18:53:27.158108 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105958fc0 con 0 2019-03-02 18:53:27.158116 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105492000 2019-03-02 18:53:27.158133 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105492000 con 0 2019-03-02 18:53:27.158140 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105794d80 2019-03-02 18:53:27.158136 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 2 0x560105958480 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158145 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105794d80 con 0 2019-03-02 18:53:27.158155 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 4 at 2019-03-02 18:53:27.156308) v1 ==== 0+0+0 (0 0 0) 0x560105795440 con 0x5601053ea000 2019-03-02 18:53:27.158171 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793707 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (1300498542 0 0) 0x560105908c00 con 0x560105677000 2019-03-02 18:53:27.158168 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793709 0x560105909b00 paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 2019-03-02 18:53:27.158182 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 2 0x560105958240 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158186 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943797 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (613287789 0 0) 0x560105908600 con 0x560105678800 2019-03-02 18:53:27.158190 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 2 0x560105492b40 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158205 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 2 0x560105794fc0 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158207 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216112 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (3257341873 0 0) 0x560105908300 con 0x56010567e000 2019-03-02 18:53:27.158216 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 5 at 2019-03-02 18:53:27.156347) v1 ==== 0+0+0 (0 0 0) 0x560105795b00 con 0x5601053ea000 2019-03-02 18:53:27.158227 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943799 0x56010500af00 paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 2019-03-02 18:53:27.158230 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 2 0x560105435d40 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158229 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793708 ==== forward(log(1 entries from seq 352 at 2019-03-02 18:53:07.138814) v1 caps allow * tid 5274 con_features 2305244844532236283) v3 ==== 345+0+0 (1198622601 0 0) 0x5601058a7400 con 0x560105677000 2019-03-02 18:53:27.158250 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 2 0x560105492000 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158245 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 2 0x560105958fc0 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158260 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943798 ==== forward(log(1 entries from seq 251 at 2019-03-02 18:53:07.139680) v1 caps allow * tid 3940 con_features 2305244844532236283) v3 ==== 345+0+0 (3144552998 0 0) 0x56010599c280 con 0x560105678800 2019-03-02 18:53:27.158282 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 2 0x560105795680 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158287 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 6 at 2019-03-02 18:53:27.156421) v1 ==== 0+0+0 (0 0 0) 0x560105795d40 con 0x5601053ea000 2019-03-02 18:53:27.158295 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 2 0x560105435200 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158300 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793709 ==== paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 ==== 84+0+0 (3782697332 0 0) 0x560105909b00 con 0x560105677000 2019-03-02 18:53:27.158307 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 2 0x560105492240 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158310 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943799 ==== paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 ==== 84+0+0 (3694817334 0 0) 0x56010500af00 con 0x560105678800 2019-03-02 18:53:27.158327 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 7 at 2019-03-02 18:53:27.156485) v1 ==== 0+0+0 (0 0 0) 0x560105992900 con 0x5601053ea000 2019-03-02 18:53:27.158342 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 8 at 2019-03-02 18:53:27.156683) v1 ==== 0+0+0 (0 0 0) 0x560105992d80 con 0x5601053ea000 2019-03-02 18:53:27.158359 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (3965368829 0 0) 0x5601054346c0 con 0x56010545a000 2019-03-02 18:53:27.158379 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 2 0x560105493440 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.158418 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 2 0x560105492480 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158453 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 2 0x560105958d80 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158475 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 2 0x560105958b40 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158491 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105534a20 0x560105992480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158501 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105992480 con 0 2019-03-02 18:53:27.158517 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1311405791 0 0) 0x5601054358c0 con 0x560105758000 2019-03-02 18:53:27.158600 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 2 0x560105992480 auth(proto 2 32 bytes epoch 0) v1 2019-03-02 18:53:27.158615 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105534a20 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158623 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/614279813 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 2019-03-02 18:53:27.158638 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (538006993 0 0) 0x560105435680 con 0x56010575b000 2019-03-02 18:53:27.158661 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 3 0x5601054346c0 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.158701 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158709 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 2019-03-02 18:53:27.158722 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (147920225 0 0) 0x560105434480 con 0x56010575c800 2019-03-02 18:53:27.158784 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158792 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 2019-03-02 18:53:27.158805 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (4035818936 0 0) 0x560105493d40 con 0x560105772000 2019-03-02 18:53:27.158873 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158881 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 2019-03-02 18:53:27.158874 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 3 0x5601054da000 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.158893 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (175811679 0 0) 0x560105435440 con 0x560105773800 2019-03-02 18:53:27.158904 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 4 0x560105794000 pool_op(delete unmanaged snap pool 116 auid 0 tid 35 name v0) v4 2019-03-02 18:53:27.158954 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105493d40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.158961 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493d40 con 0 2019-03-02 18:53:27.158975 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2028020967 0 0) 0x560105492900 con 0x560105775000 2019-03-02 18:53:27.159054 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159062 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/94488736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 2019-03-02 18:53:27.159075 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (930740453 0 0) 0x560105958240 con 0x560105776800 2019-03-02 18:53:27.159086 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 3 0x560105493d40 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.159153 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b8c0 0x560105492900 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159161 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3202181189 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492900 con 0 2019-03-02 18:53:27.159175 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1212249513 0 0) 0x560105958480 con 0x560105788000 2019-03-02 18:53:27.159252 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059845a0 0x560105958240 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159254 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 3 0x560105435680 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.159262 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3953751394 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958240 con 0 2019-03-02 18:53:27.159269 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 3 0x5601056d8600 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.159283 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3296193308 0 0) 0x5601054938c0 con 0x560105789800 2019-03-02 18:53:27.159341 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 3 0x5601054358c0 auth(proto 2 181 bytes epoch 0) v1 2019-03-02 18:53:27.159384 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984480 0x560105958480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159392 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/1267131636 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958480 con 0 2019-03-02 18:53:27.159407 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3966250534 0 0) 0x560105435d40 con 0x5601058f8000 2019-03-02 18:53:27.159466 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 3 0x560105435440 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.159486 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 2 0x560105434480 auth(proto 2 144 bytes epoch 0) v1 2019-03-02 18:53:27.159488 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a7e0 0x5601054938c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159493 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3482224786 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054938c0 con 0 2019-03-02 18:53:27.159508 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3378494650 0 0) 0x560105492000 con 0x5601058f9800 2019-03-02 18:53:27.159584 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b9e0 0x560105435d40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159592 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3137663203 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435d40 con 0 2019-03-02 18:53:27.159606 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1061402963 0 0) 0x560105493200 con 0x5601058fb000 2019-03-02 18:53:27.159598 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 3 0x560105252400 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.159689 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059847e0 0x560105492000 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159697 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.33:0/140057803 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492000 con 0 2019-03-02 18:53:27.159711 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1382660310 0 0) 0x560105493440 con 0x5601058fc800 2019-03-02 18:53:27.159786 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535200 0x560105493200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159794 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.35:0/3274667345 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493200 con 0 2019-03-02 18:53:27.159820 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1173792373 0 0) 0x560105492240 con 0x56010543b800 2019-03-02 18:53:27.159821 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 3 0x560105253000 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.159871 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 3 0x5601056d9e00 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.159910 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bc20 0x560105493440 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.159918 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/1733074246 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493440 con 0 2019-03-02 18:53:27.159932 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (202713985 0 0) 0x560105795680 con 0x56010593e000 2019-03-02 18:53:27.160010 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535d40 0x560105492240 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160017 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/2175704180 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492240 con 0 2019-03-02 18:53:27.160031 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (799206973 0 0) 0x560105492480 con 0x56010545b800 2019-03-02 18:53:27.160092 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160100 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 2019-03-02 18:53:27.160113 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2241480435 0 0) 0x5601054926c0 con 0x56010593f800 2019-03-02 18:53:27.160152 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 3 0x560105252c00 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160172 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 3 0x5601056d6000 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160184 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105492480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160193 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492480 con 0 2019-03-02 18:53:27.160206 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1127919766 0 0) 0x560105958b40 con 0x560105941000 2019-03-02 18:53:27.160212 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 3 0x5601054da200 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160270 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160285 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 2019-03-02 18:53:27.160302 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (3290125480 0 0) 0x560105958d80 con 0x560105942800 2019-03-02 18:53:27.160350 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 3 0x5601054926c0 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.160366 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105958b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160370 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 3 0x560105253800 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160374 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958b40 con 0 2019-03-02 18:53:27.160387 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1138593506 0 0) 0x560105492fc0 con 0x56010595c000 2019-03-02 18:53:27.160447 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105958d80 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160454 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958d80 con 0 2019-03-02 18:53:27.160468 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2443371689 0 0) 0x560105958fc0 con 0x56010595d800 2019-03-02 18:53:27.160546 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105492fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160554 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/683377982 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492fc0 con 0 2019-03-02 18:53:27.160567 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2901108871 0 0) 0x560105794fc0 con 0x56010595f000 2019-03-02 18:53:27.160563 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 3 0x5601056d7800 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160590 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 3 0x560105492fc0 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.160640 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 3 0x5601056d9a00 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.160654 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535320 0x560105958fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160663 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/1251637173 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958fc0 con 0 2019-03-02 18:53:27.160664 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 3 0x560105795680 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.160677 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (558639228 0 0) 0x560105435200 con 0x560105960800 2019-03-02 18:53:27.160755 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059846c0 0x560105794fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160763 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/419113882 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794fc0 con 0 2019-03-02 18:53:27.160776 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (102187718 0 0) 0x560105492b40 con 0x560105972000 2019-03-02 18:53:27.160779 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 3 0x5601056d8800 mon_subscribe({monmap=15+}) v2 2019-03-02 18:53:27.160841 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105435200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160859 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435200 con 0 2019-03-02 18:53:27.160864 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 3 0x560105958d80 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.160876 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2355473505 0 0) 0x560105992480 con 0x560105973800 2019-03-02 18:53:27.160882 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 3 0x5601056d7e00 mon_subscribe({monmap=15+}) v2 2019-03-02 18:53:27.160925 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 3 0x560105958b40 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.160942 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.160950 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 2019-03-02 18:53:27.160965 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (199693358 0 0) 0x560105434fc0 con 0x56010548f000 2019-03-02 18:53:27.161024 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 3 0x560105492b40 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.161026 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 3 0x5601056d7400 mon_subscribe({monmap=15+}) v2 2019-03-02 18:53:27.161043 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105992480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161049 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3589307736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105992480 con 0 2019-03-02 18:53:27.161063 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1048373748 0 0) 0x560105493b00 con 0x560105490800 2019-03-02 18:53:27.161135 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a6c0 0x560105434fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161143 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3361644023 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434fc0 con 0 2019-03-02 18:53:27.161156 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1007212614 0 0) 0x560105795200 con 0x56010563b000 2019-03-02 18:53:27.161216 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984240 0x560105493b00 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161224 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493b00 con 0 2019-03-02 18:53:27.161237 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (2612887804 0 0) 0x5601054346c0 con 0x56010545a000 2019-03-02 18:53:27.161280 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 3 0x5601056d9200 mon_subscribe({monmap=15+}) v2 2019-03-02 18:53:27.161352 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984240 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161349 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 3 0x5601056d5000 mon_subscribe({monmap=15+}) v2 2019-03-02 18:53:27.161361 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 2019-03-02 18:53:27.161375 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601054da000 con 0x560105758000 2019-03-02 18:53:27.161398 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (4024092099 0 0) 0x560105435680 con 0x56010575b000 2019-03-02 18:53:27.161497 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bd40 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161504 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 2019-03-02 18:53:27.161517 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 3 ==== auth(proto 2 181 bytes epoch 0) v1 ==== 211+0+0 (1957492419 0 0) 0x5601054358c0 con 0x56010575c800 2019-03-02 18:53:27.161530 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 3 0x560105795200 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.161561 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 4 0x560104ff7a00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.161582 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 5 0x5601056d5e00 osd_beacon(pgs [109.29,109.9,109.65,109.69] lec 79198 v79198) v1 2019-03-02 18:53:27.161626 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535560 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161633 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 2019-03-02 18:53:27.161647 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1498713863 0 0) 0x560105435440 con 0x560105772000 2019-03-02 18:53:27.161737 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535560 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161744 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 2019-03-02 18:53:27.161756 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1354212678 0 0) 0x560105493d40 con 0x560105773800 2019-03-02 18:53:27.161774 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 3 0x560105493b00 auth(proto 2 165 bytes epoch 0) v1 2019-03-02 18:53:27.161843 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105545560 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161849 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 2019-03-02 18:53:27.161862 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d8600 con 0x560105775000 2019-03-02 18:53:27.161876 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105253000 con 0x560105776800 2019-03-02 18:53:27.161886 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d9e00 con 0x560105788000 2019-03-02 18:53:27.161896 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105252400 con 0x560105789800 2019-03-02 18:53:27.161906 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.7893124 192.168.253.44:0/219447902 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1800022541 0 0) 0x560105434480 con 0x56010578b000 2019-03-02 18:53:27.161982 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535440 0x560105493d40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.161990 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.44:0/219447902 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493d40 con 0 2019-03-02 18:53:27.162004 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105252c00 con 0x5601058f8000 2019-03-02 18:53:27.162009 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 4 0x5601056d6c00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162018 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601054da200 con 0x5601058f9800 2019-03-02 18:53:27.162027 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105253800 con 0x5601058fb000 2019-03-02 18:53:27.162030 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 5 0x5601056d6a00 osd_beacon(pgs [109.44,109.5c,109.37,109.4a,109.74,109.5f] lec 79198 v79198) v1 2019-03-02 18:53:27.162036 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d6000 con 0x5601058fc800 2019-03-02 18:53:27.162042 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 6 0x5601056d9e00 osd_beacon(pgs [] lec 79198 v79198) v1 2019-03-02 18:53:27.162045 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d7800 con 0x56010543b800 2019-03-02 18:53:27.162052 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d9a00 con 0x56010593e000 2019-03-02 18:53:27.162060 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1446845268 0 0) 0x560105795680 con 0x56010545b800 2019-03-02 18:53:27.162082 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 4 0x5601056d8600 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162097 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 5 0x560105253000 osd_beacon(pgs [105.c,105.1c8,116.78,105.109,105.d7,105.64,105.297,85.ff,116.17,115.10,105.108,85.6f,85.a7,105.2e0,105.329,105.2ae,105.253,119.37,85.aa,105.19f,105.d6,105.73,115.13,85.1a,105.310,85.98,105.6f,105.97,85.6,121.6f,116.e6,105.1f8,105.31f,105.2ea,105.1d5,105.100,121.30,105.105,121.56,85.7a,105.e4,121.79,121.18,105.18c,108.1d,116.1a,121.33,119.3d,105.3c,116.f0,105.261,105.21a,85.b0,105.1f4,116.7b,85.84,105.4,85.38,115.c,119.8,105.30,105.11d,121.3e,108.2b,105.1bc,105.190,105.36,116.99,105.290,85.18,85.e6,116.c7,116.4b,105.2fd,85.a5,115.e,105.16b,116.f6,105.3f7,85.8a] lec 79197 v79198) v1 2019-03-02 18:53:27.162152 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105545680 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162160 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 2019-03-02 18:53:27.162173 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3437510617 0 0) 0x5601054926c0 con 0x56010593f800 2019-03-02 18:53:27.162259 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055457a0 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162265 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 2019-03-02 18:53:27.162293 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 4 0x560105252400 mon_subscribe({fsmap=2+,log-info=0+,mgrdigest=0+,mgrmap=0+,monmap=15+,osdmap=79199,servicemap=0+}) v2 2019-03-02 18:53:27.162314 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1117087331 0 0) 0x560105958d80 con 0x560105941000 2019-03-02 18:53:27.162316 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 5 0x56010598fb00 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 2019-03-02 18:53:27.162335 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 6 0x56010599cf00 monmgrreport(0 checks) v1 2019-03-02 18:53:27.162352 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 7 0x56010598f800 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 2019-03-02 18:53:27.162361 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 4 0x5601056d9800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162367 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 8 0x5601058a7180 monmgrreport(0 checks) v1 2019-03-02 18:53:27.162388 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 9 0x56010598f500 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 2019-03-02 18:53:27.162385 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 5 0x5601056d4a00 osd_beacon(pgs [121.5f,121.22,121.f,105.2b7,121.4f,105.2e2,105.110,105.db,119.10,119.2b,105.ee,105.2cb,105.2b,105.86,105.3b,105.2ce,105.3b4,85.50,105.d1,115.8,115.32,85.d9,105.ca,105.ef,108.0,105.fa,119.21,85.3,119.20,105.c0,116.3c,105.80,108.4e,85.bc,105.1dc,105.148,116.df,116.a1,116.82,105.3fd,116.63,116.a4,105.6d,105.0,105.19e,116.4d,105.18f,116.d3,121.3c,105.33a,116.a8,108.79,105.8c,85.4b,105.1cb,105.33f,105.1b6,105.3ab,105.188,105.2dc,108.3c,105.132] lec 79197 v79198) v1 2019-03-02 18:53:27.162403 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 6 0x5601056d4200 osd_beacon(pgs [] lec 79197 v79198) v1 2019-03-02 18:53:27.162407 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 10 0x56010599d400 monmgrreport(0 checks) v1 2019-03-02 18:53:27.162411 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535680 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162416 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 2019-03-02 18:53:27.162428 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (2825895776 0 0) 0x560105958b40 con 0x560105942800 2019-03-02 18:53:27.162516 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055358c0 0x560105958d80 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162523 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958d80 con 0 2019-03-02 18:53:27.162535 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3530128003 0 0) 0x560105492fc0 con 0x56010595c000 2019-03-02 18:53:27.162563 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 4 0x5601056d7200 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162575 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 5 0x5601056d5600 osd_beacon(pgs [109.2b,109.1f,109.76,109.28,109.1e,109.5] lec 79198 v79198) v1 2019-03-02 18:53:27.162624 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055359e0 0x560105958b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162630 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958b40 con 0 2019-03-02 18:53:27.162643 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d8800 con 0x56010595d800 2019-03-02 18:53:27.162655 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d7e00 con 0x56010595f000 2019-03-02 18:53:27.162665 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d7400 con 0x560105960800 2019-03-02 18:53:27.162674 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1436488630 0 0) 0x560105492b40 con 0x560105972000 2019-03-02 18:53:27.162776 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055357a0 0x560105492fc0 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162775 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 11 0x560105909800 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 2019-03-02 18:53:27.162780 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 4 0x5601056d7800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162786 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492fc0 con 0 2019-03-02 18:53:27.162797 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 12 0x56010599db80 monmgrreport(0 checks) v1 2019-03-02 18:53:27.162799 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1132521040 0 0) 0x560105795200 con 0x560105973800 2019-03-02 18:53:27.162798 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 5 0x5601056d7e00 osd_beacon(pgs [121.47,121.29,121.e,105.223,105.298,105.4b,116.91,105.7e,105.170,105.3b7,105.137,105.27b,105.368,85.17,105.226,108.1,105.11a,105.122,105.2f3,105.85,115.2a,105.18e,105.27,85.10,85.54,105.192,85.37,116.6,108.7a,85.97,116.a2,105.370,116.65,116.cf,119.23,116.d4,108.3e,121.7d,85.df,105.e3,105.138,85.e2,105.20f,108.56,85.ab,85.62,105.1b9,105.e,105.26c,105.294,115.3a,105.246,105.2e4,105.1bd,105.28e,105.d9,105.134,105.3b5,105.23a,105.14b,105.293,85.12,105.3d9,105.210] lec 79197 v79198) v1 2019-03-02 18:53:27.162810 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 4 0x5601056d4000 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.162818 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 5 0x560105252c00 osd_beacon(pgs [109.52,109.3d] lec 79198 v79198) v1 2019-03-02 18:53:27.162842 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 3 0x5601056d4600 mon_subscribe({monmap=15+,osdmap=79199}) v2 2019-03-02 18:53:27.162891 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055458c0 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.162898 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 2019-03-02 18:53:27.162912 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d9200 con 0x56010548f000 2019-03-02 18:53:27.162925 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d5000 con 0x560105490800 2019-03-02 18:53:27.162935 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (399444511 0 0) 0x560105493b00 con 0x56010563b000 2019-03-02 18:53:27.162987 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 4 0x5601056d7400 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.163009 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 5 0x5601056d8800 osd_beacon(pgs [109.71,109.66,109.31,109.41] lec 79198 v79198) v1 2019-03-02 18:53:27.163017 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 4 0x560105253800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.163023 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 6 0x5601056d5800 osd_beacon(pgs [] lec 79198 v79198) v1 2019-03-02 18:53:27.163029 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 5 0x5601054da200 osd_beacon(pgs [121.50,121.8,105.e2,121.74,105.1f2,105.346,105.2,105.1ee,105.24a,105.1af,105.187,105.302,105.3bd,121.5d,108.7c,105.dc,116.d9,105.1e2,116.ba,85.d1,116.5c,85.7d,105.cf,105.180,116.2f,116.46,115.0,116.66,85.47,105.2f8,116.d8,116.ed,105.379,108.5,105.ea,105.1d9,116.9d,116.f4,105.3a,85.e1,116.bb,108.4a,116.fb,121.31,115.3b,116.45,105.2ef,105.34f,85.1c,85.96,105.1ea,116.c8,105.263,116.53,105.32a,108.22,105.c6,108.77,105.397,105.348,108.1a] lec 79197 v79198) v1 2019-03-02 18:53:27.163034 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055459e0 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 2019-03-02 18:53:27.163046 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 6 0x5601054da000 osd_beacon(pgs [] lec 79197 v79198) v1 2019-03-02 18:53:27.163044 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 2019-03-02 18:53:27.163061 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 4 ==== mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 ==== 69+0+0 (2557949307 0 0) 0x560104ff7a00 con 0x56010545a000 2019-03-02 18:53:27.163095 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- mgrmap(e 519) v1 -- 0x5601056b02c0 con 0 2019-03-02 18:53:27.163113 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 4 0x5601054da600 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.163112 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 4 ==== pool_op(delete unmanaged snap pool 116 auid 0 tid 35 name v0) v4 ==== 65+0+0 (3422141634 0 0) 0x560105794000 con 0x560105758000 2019-03-02 18:53:27.163129 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 5 0x5601054dba00 osd_beacon(pgs [121.77,105.12d,116.5f,85.a4,85.14,85.b1,116.a,85.4d,116.8a,105.177,115.24,105.3f3,105.336,85.60,105.232,108.7d,85.75,108.5f,85.ee,105.65,105.1fc,105.3b9,116.de,105.c3,108.9,85.f4,105.22c,105.60,116.ad,105.39c,108.58,121.6a,116.67,121.3b,116.9f,105.1db,85.74,105.23,105.36b,105.211,116.e3,105.289,105.12e,105.7f,115.1a,121.51,105.35d,116.76,105.392,105.a0,105.3fc,108.20,85.ac,85.eb,105.10b,105.3d5,121.c,105.26a,105.3e3,85.b6,105.1ca,105.3d6,108.39,105.388,116.ec,115.3d,105.3b0,105.145,105.2ac,115.5,116.30,108.28] lec 79197 v79198) v1 2019-03-02 18:53:27.163517 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 4 0x5601056d9a00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.163537 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 5 0x5601056d4c00 osd_beacon(pgs [121.7b,121.5c,85.ca,105.295,105.2b0,105.1a3,105.327,105.304,105.3e6,105.1e0,105.19b,85.90,105.81,85.1f,105.2bc,119.18,105.1c7,105.2f4,105.21d,105.3af,105.3f2,116.26,116.27,116.c0,85.e7,105.39e,105.44,116.5,105.143,116.1c,116.ae,105.2e6,105.26d,116.b0,105.1b3,105.3cb,105.d2,108.1c,105.2a3,116.75,105.d4,105.279,105.37c,105.a9,105.a3,85.56,105.3a4,85.63,85.c2,105.74,105.31b,116.6d,105.34e,105.212,108.65,105.1fe] lec 79197 v79198) v1 2019-03-02 18:53:27.163637 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 4 0x5601056d9c00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 2019-03-02 18:53:27.163654 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 5 0x5601056d6e00 osd_beacon(pgs [121.7a,121.3a,121.d,115.2e,105.fe,105.2b6,105.2b2,105.9b,85.8b,119.0,105.272,105.3dd,116.60,105.51,105.372,105.29e,105.350,85.53,105.331,105.28b,85.4,105.9f,116.ff,85.15,121.5,116.8,115.f,108.10,116.7a,105.158,105.213,116.b7,105.238,116.16,105.333,85.fb,116.e8,105.1ed,85.9f,105.31d,105.1b2,105.224,85.8f,108.15,116.d,105.be,85.24,115.21,105.173,105.369,85.f1,105.364,105.236,105.f7,85.92,108.35,119.1f,105.296,105.11e,105.2ca,105.167,105.de,105.334,116.2a,105.3c4,105.3e7,105.40] lec 79197 v79198) v1 2019-03-02 18:53:27.163666 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 6 0x5601056d6800 osd_beacon(pgs [] lec 79197 v79198) v1 2019-03-02 18:53:27.166322 7fe359f02700 -1 *** Caught signal (Aborted) ** in thread 7fe359f02700 thread_name:ms_dispatch ceph version 12.2.5-2-ge988fb6 (e988fb6c5457b9dd6f1e11ce1b5fd58b4c3a828c) luminous (stable) 1: (()+0x8f53b1) [0x5600fb5d83b1] 2: (()+0xf6d0) [0x7fe3634736d0] 3: (gsignal()+0x37) [0x7fe3607ae277] 4: (abort()+0x148) [0x7fe3607af968] 5: (std::_Rb_tree_iterator, std::allocator > > > > std::_Rb_tree, std::allocator > > >, std::_Select1st, std::allocator > > > >, std::less, std::allocator, std::allocator > > > > >::_M_emplace_hint_unique, std::allocator > > > >(std::_Rb_tree_const_iterator, std::allocator > > > >, std::pair, std::allocator > > >&&)+0) [0x5600fb420780] 6: (pg_pool_t::remove_unmanaged_snap(snapid_t)+0x4d) [0x5600fb40d2dd] 7: (OSDMonitor::prepare_pool_op(boost::intrusive_ptr)+0xe40) [0x5600fb200f10] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x336) [0x5600fb2388c6] 9: (PaxosService::dispatch(boost::intrusive_ptr)+0xaf8) [0x5600fb1c4858] 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x5af) [0x5600fb0a4fef] 11: (Monitor::_ms_dispatch(Message*)+0x7eb) [0x5600fb0a65db] 12: (Monitor::ms_dispatch(Message*)+0x23) [0x5600fb0d2743] 13: (DispatchQueue::entry()+0x792) [0x5600fb583752] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x5600fb37c90d] 15: (()+0x7e25) [0x7fe36346be25] 16: (clone()+0x6d) [0x7fe360876bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -1212> 2019-03-02 18:53:17.940174 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perfcounters_dump hook 0x560104fac190 -1211> 2019-03-02 18:53:17.940191 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command 1 hook 0x560104fac190 -1210> 2019-03-02 18:53:17.940195 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perf dump hook 0x560104fac190 -1209> 2019-03-02 18:53:17.940200 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perfcounters_schema hook 0x560104fac190 -1208> 2019-03-02 18:53:17.940205 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perf histogram dump hook 0x560104fac190 -1207> 2019-03-02 18:53:17.940209 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command 2 hook 0x560104fac190 -1206> 2019-03-02 18:53:17.940212 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perf schema hook 0x560104fac190 -1205> 2019-03-02 18:53:17.940216 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perf histogram schema hook 0x560104fac190 -1204> 2019-03-02 18:53:17.940225 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command perf reset hook 0x560104fac190 -1203> 2019-03-02 18:53:17.940228 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config show hook 0x560104fac190 -1202> 2019-03-02 18:53:17.940231 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config help hook 0x560104fac190 -1201> 2019-03-02 18:53:17.940235 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config set hook 0x560104fac190 -1200> 2019-03-02 18:53:17.940239 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config get hook 0x560104fac190 -1199> 2019-03-02 18:53:17.940243 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config diff hook 0x560104fac190 -1198> 2019-03-02 18:53:17.940247 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command config diff get hook 0x560104fac190 -1197> 2019-03-02 18:53:17.940251 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command log flush hook 0x560104fac190 -1196> 2019-03-02 18:53:17.940254 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command log dump hook 0x560104fac190 -1195> 2019-03-02 18:53:17.940258 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command log reopen hook 0x560104fac190 -1194> 2019-03-02 18:53:17.940267 7fe3640c4ec0 5 asok(0x560104ffa1c0) register_command dump_mempools hook 0x5601051c4e28 -1193> 2019-03-02 18:53:17.948283 7fe3640c4ec0 0 set uid:gid to 1001:1001 (ceph:ceph) -1192> 2019-03-02 18:53:17.948303 7fe3640c4ec0 0 ceph version 12.2.5-2-ge988fb6 (e988fb6c5457b9dd6f1e11ce1b5fd58b4c3a828c) luminous (stable), process (unknown), pid 3016783 -1191> 2019-03-02 18:53:17.948362 7fe3640c4ec0 0 pidfile_write: ignore empty --pid-file -1190> 2019-03-02 18:53:17.980882 7fe3640c4ec0 0 load: jerasure load: lrc load: isa -1189> 2019-03-02 18:53:17.981041 7fe3640c4ec0 0 set rocksdb option compression = kNoCompression -1188> 2019-03-02 18:53:17.981052 7fe3640c4ec0 0 set rocksdb option write_buffer_size = 33554432 -1187> 2019-03-02 18:53:17.981077 7fe3640c4ec0 0 set rocksdb option compression = kNoCompression -1186> 2019-03-02 18:53:17.981084 7fe3640c4ec0 0 set rocksdb option write_buffer_size = 33554432 -1185> 2019-03-02 18:53:17.981256 7fe3640c4ec0 4 rocksdb: RocksDB version: 5.4.0 -1184> 2019-03-02 18:53:17.981267 7fe3640c4ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -1183> 2019-03-02 18:53:17.981269 7fe3640c4ec0 4 rocksdb: Compile date Aug 6 2018 -1182> 2019-03-02 18:53:17.981279 7fe3640c4ec0 4 rocksdb: DB SUMMARY -1181> 2019-03-02 18:53:17.981331 7fe3640c4ec0 4 rocksdb: CURRENT file: CURRENT -1180> 2019-03-02 18:53:17.981336 7fe3640c4ec0 4 rocksdb: IDENTITY file: IDENTITY -1179> 2019-03-02 18:53:17.981344 7fe3640c4ec0 4 rocksdb: MANIFEST file: MANIFEST-000259 size: 288 Bytes -1178> 2019-03-02 18:53:17.981347 7fe3640c4ec0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-storage01roc/store.db dir, Total Num: 4, files: 000250.sst 000252.sst 000255.sst 000258.sst -1177> 2019-03-02 18:53:17.981351 7fe3640c4ec0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-storage01roc/store.db: 000260.log size: 49357 ; -1176> 2019-03-02 18:53:17.981354 7fe3640c4ec0 4 rocksdb: Options.error_if_exists: 0 -1175> 2019-03-02 18:53:17.981355 7fe3640c4ec0 4 rocksdb: Options.create_if_missing: 0 -1174> 2019-03-02 18:53:17.981356 7fe3640c4ec0 4 rocksdb: Options.paranoid_checks: 1 -1173> 2019-03-02 18:53:17.981360 7fe3640c4ec0 4 rocksdb: Options.env: 0x560103e0c140 -1172> 2019-03-02 18:53:17.981361 7fe3640c4ec0 4 rocksdb: Options.info_log: 0x56010524ef80 -1171> 2019-03-02 18:53:17.981362 7fe3640c4ec0 4 rocksdb: Options.max_open_files: -1 -1170> 2019-03-02 18:53:17.981363 7fe3640c4ec0 4 rocksdb: Options.max_file_opening_threads: 16 -1169> 2019-03-02 18:53:17.981364 7fe3640c4ec0 4 rocksdb: Options.use_fsync: 0 -1168> 2019-03-02 18:53:17.981365 7fe3640c4ec0 4 rocksdb: Options.max_log_file_size: 0 -1167> 2019-03-02 18:53:17.981366 7fe3640c4ec0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -1166> 2019-03-02 18:53:17.981368 7fe3640c4ec0 4 rocksdb: Options.log_file_time_to_roll: 0 -1165> 2019-03-02 18:53:17.981369 7fe3640c4ec0 4 rocksdb: Options.keep_log_file_num: 1000 -1164> 2019-03-02 18:53:17.981370 7fe3640c4ec0 4 rocksdb: Options.recycle_log_file_num: 0 -1163> 2019-03-02 18:53:17.981371 7fe3640c4ec0 4 rocksdb: Options.allow_fallocate: 1 -1162> 2019-03-02 18:53:17.981372 7fe3640c4ec0 4 rocksdb: Options.allow_mmap_reads: 0 -1161> 2019-03-02 18:53:17.981373 7fe3640c4ec0 4 rocksdb: Options.allow_mmap_writes: 0 -1160> 2019-03-02 18:53:17.981374 7fe3640c4ec0 4 rocksdb: Options.use_direct_reads: 0 -1159> 2019-03-02 18:53:17.981375 7fe3640c4ec0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -1158> 2019-03-02 18:53:17.981377 7fe3640c4ec0 4 rocksdb: Options.create_missing_column_families: 0 -1157> 2019-03-02 18:53:17.981378 7fe3640c4ec0 4 rocksdb: Options.db_log_dir: -1156> 2019-03-02 18:53:17.981379 7fe3640c4ec0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-storage01roc/store.db -1155> 2019-03-02 18:53:17.981380 7fe3640c4ec0 4 rocksdb: Options.table_cache_numshardbits: 6 -1154> 2019-03-02 18:53:17.981380 7fe3640c4ec0 4 rocksdb: Options.max_subcompactions: 1 -1153> 2019-03-02 18:53:17.981381 7fe3640c4ec0 4 rocksdb: Options.max_background_flushes: 1 -1152> 2019-03-02 18:53:17.981381 7fe3640c4ec0 4 rocksdb: Options.WAL_ttl_seconds: 0 -1151> 2019-03-02 18:53:17.981382 7fe3640c4ec0 4 rocksdb: Options.WAL_size_limit_MB: 0 -1150> 2019-03-02 18:53:17.981383 7fe3640c4ec0 4 rocksdb: Options.manifest_preallocation_size: 4194304 -1149> 2019-03-02 18:53:17.981383 7fe3640c4ec0 4 rocksdb: Options.is_fd_close_on_exec: 1 -1148> 2019-03-02 18:53:17.981384 7fe3640c4ec0 4 rocksdb: Options.advise_random_on_open: 1 -1147> 2019-03-02 18:53:17.981385 7fe3640c4ec0 4 rocksdb: Options.db_write_buffer_size: 0 -1146> 2019-03-02 18:53:17.981385 7fe3640c4ec0 4 rocksdb: Options.access_hint_on_compaction_start: 1 -1145> 2019-03-02 18:53:17.981386 7fe3640c4ec0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 -1144> 2019-03-02 18:53:17.981386 7fe3640c4ec0 4 rocksdb: Options.compaction_readahead_size: 0 -1143> 2019-03-02 18:53:17.981387 7fe3640c4ec0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -1142> 2019-03-02 18:53:17.981388 7fe3640c4ec0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 -1141> 2019-03-02 18:53:17.981388 7fe3640c4ec0 4 rocksdb: Options.use_adaptive_mutex: 0 -1140> 2019-03-02 18:53:17.981389 7fe3640c4ec0 4 rocksdb: Options.rate_limiter: (nil) -1139> 2019-03-02 18:53:17.981390 7fe3640c4ec0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -1138> 2019-03-02 18:53:17.981391 7fe3640c4ec0 4 rocksdb: Options.bytes_per_sync: 0 -1137> 2019-03-02 18:53:17.981391 7fe3640c4ec0 4 rocksdb: Options.wal_bytes_per_sync: 0 -1136> 2019-03-02 18:53:17.981392 7fe3640c4ec0 4 rocksdb: Options.wal_recovery_mode: 2 -1135> 2019-03-02 18:53:17.981392 7fe3640c4ec0 4 rocksdb: Options.enable_thread_tracking: 0 -1134> 2019-03-02 18:53:17.981393 7fe3640c4ec0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -1133> 2019-03-02 18:53:17.981394 7fe3640c4ec0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -1132> 2019-03-02 18:53:17.981394 7fe3640c4ec0 4 rocksdb: Options.write_thread_max_yield_usec: 100 -1131> 2019-03-02 18:53:17.981395 7fe3640c4ec0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -1130> 2019-03-02 18:53:17.981403 7fe3640c4ec0 4 rocksdb: Options.row_cache: None -1129> 2019-03-02 18:53:17.981404 7fe3640c4ec0 4 rocksdb: Options.wal_filter: None -1128> 2019-03-02 18:53:17.981405 7fe3640c4ec0 4 rocksdb: Options.avoid_flush_during_recovery: 0 -1127> 2019-03-02 18:53:17.981405 7fe3640c4ec0 4 rocksdb: Options.base_background_compactions: 1 -1126> 2019-03-02 18:53:17.981406 7fe3640c4ec0 4 rocksdb: Options.max_background_compactions: 1 -1125> 2019-03-02 18:53:17.981406 7fe3640c4ec0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -1124> 2019-03-02 18:53:17.981407 7fe3640c4ec0 4 rocksdb: Options.delayed_write_rate : 16777216 -1123> 2019-03-02 18:53:17.981408 7fe3640c4ec0 4 rocksdb: Options.max_total_wal_size: 0 -1122> 2019-03-02 18:53:17.981408 7fe3640c4ec0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -1121> 2019-03-02 18:53:17.981409 7fe3640c4ec0 4 rocksdb: Options.stats_dump_period_sec: 600 -1120> 2019-03-02 18:53:17.981410 7fe3640c4ec0 4 rocksdb: Compression algorithms supported: -1119> 2019-03-02 18:53:17.981412 7fe3640c4ec0 4 rocksdb: Snappy supported: 0 -1118> 2019-03-02 18:53:17.981413 7fe3640c4ec0 4 rocksdb: Zlib supported: 0 -1117> 2019-03-02 18:53:17.981413 7fe3640c4ec0 4 rocksdb: Bzip supported: 0 -1116> 2019-03-02 18:53:17.981414 7fe3640c4ec0 4 rocksdb: LZ4 supported: 0 -1115> 2019-03-02 18:53:17.981415 7fe3640c4ec0 4 rocksdb: ZSTD supported: 0 -1114> 2019-03-02 18:53:17.981415 7fe3640c4ec0 4 rocksdb: Fast CRC32 supported: 1 -1113> 2019-03-02 18:53:17.981735 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000259 -1112> 2019-03-02 18:53:17.981852 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: -1111> 2019-03-02 18:53:17.981858 7fe3640c4ec0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -1110> 2019-03-02 18:53:17.981865 7fe3640c4ec0 4 rocksdb: Options.merge_operator: -1109> 2019-03-02 18:53:17.981866 7fe3640c4ec0 4 rocksdb: Options.compaction_filter: None -1108> 2019-03-02 18:53:17.981866 7fe3640c4ec0 4 rocksdb: Options.compaction_filter_factory: None -1107> 2019-03-02 18:53:17.981867 7fe3640c4ec0 4 rocksdb: Options.memtable_factory: SkipListFactory -1106> 2019-03-02 18:53:17.981868 7fe3640c4ec0 4 rocksdb: Options.table_factory: BlockBasedTable -1105> 2019-03-02 18:53:17.981885 7fe3640c4ec0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560104faa168) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x560105067278 block_cache_name: LRUCache block_cache_options: capacity : 134217728 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 -1104> 2019-03-02 18:53:17.981894 7fe3640c4ec0 4 rocksdb: Options.write_buffer_size: 33554432 -1103> 2019-03-02 18:53:17.981895 7fe3640c4ec0 4 rocksdb: Options.max_write_buffer_number: 2 -1102> 2019-03-02 18:53:17.981896 7fe3640c4ec0 4 rocksdb: Options.compression: NoCompression -1101> 2019-03-02 18:53:17.981897 7fe3640c4ec0 4 rocksdb: Options.bottommost_compression: Disabled -1100> 2019-03-02 18:53:17.981898 7fe3640c4ec0 4 rocksdb: Options.prefix_extractor: nullptr -1099> 2019-03-02 18:53:17.981899 7fe3640c4ec0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -1098> 2019-03-02 18:53:17.981899 7fe3640c4ec0 4 rocksdb: Options.num_levels: 7 -1097> 2019-03-02 18:53:17.981900 7fe3640c4ec0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -1096> 2019-03-02 18:53:17.981901 7fe3640c4ec0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -1095> 2019-03-02 18:53:17.981901 7fe3640c4ec0 4 rocksdb: Options.compression_opts.window_bits: -14 -1094> 2019-03-02 18:53:17.981902 7fe3640c4ec0 4 rocksdb: Options.compression_opts.level: -1 -1093> 2019-03-02 18:53:17.981903 7fe3640c4ec0 4 rocksdb: Options.compression_opts.strategy: 0 -1092> 2019-03-02 18:53:17.981903 7fe3640c4ec0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -1091> 2019-03-02 18:53:17.981904 7fe3640c4ec0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -1090> 2019-03-02 18:53:17.981905 7fe3640c4ec0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -1089> 2019-03-02 18:53:17.981905 7fe3640c4ec0 4 rocksdb: Options.level0_stop_writes_trigger: 36 -1088> 2019-03-02 18:53:17.981906 7fe3640c4ec0 4 rocksdb: Options.target_file_size_base: 67108864 -1087> 2019-03-02 18:53:17.981907 7fe3640c4ec0 4 rocksdb: Options.target_file_size_multiplier: 1 -1086> 2019-03-02 18:53:17.981907 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -1085> 2019-03-02 18:53:17.981908 7fe3640c4ec0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -1084> 2019-03-02 18:53:17.981909 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -1083> 2019-03-02 18:53:17.981911 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -1082> 2019-03-02 18:53:17.981915 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -1081> 2019-03-02 18:53:17.981916 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -1080> 2019-03-02 18:53:17.981917 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -1079> 2019-03-02 18:53:17.981921 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -1078> 2019-03-02 18:53:17.981922 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -1077> 2019-03-02 18:53:17.981922 7fe3640c4ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -1076> 2019-03-02 18:53:17.981923 7fe3640c4ec0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -1075> 2019-03-02 18:53:17.981923 7fe3640c4ec0 4 rocksdb: Options.max_compaction_bytes: 1677721600 -1074> 2019-03-02 18:53:17.981924 7fe3640c4ec0 4 rocksdb: Options.arena_block_size: 4194304 -1073> 2019-03-02 18:53:17.981925 7fe3640c4ec0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -1072> 2019-03-02 18:53:17.981926 7fe3640c4ec0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -1071> 2019-03-02 18:53:17.981930 7fe3640c4ec0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -1070> 2019-03-02 18:53:17.981931 7fe3640c4ec0 4 rocksdb: Options.disable_auto_compactions: 0 -1069> 2019-03-02 18:53:17.981932 7fe3640c4ec0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -1068> 2019-03-02 18:53:17.981933 7fe3640c4ec0 4 rocksdb: Options.compaction_pri: kByCompensatedSize -1067> 2019-03-02 18:53:17.981934 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -1066> 2019-03-02 18:53:17.981934 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -1065> 2019-03-02 18:53:17.981935 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -1064> 2019-03-02 18:53:17.981936 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -1063> 2019-03-02 18:53:17.981939 7fe3640c4ec0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -1062> 2019-03-02 18:53:17.981940 7fe3640c4ec0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -1061> 2019-03-02 18:53:17.981941 7fe3640c4ec0 4 rocksdb: Options.table_properties_collectors: -1060> 2019-03-02 18:53:17.981942 7fe3640c4ec0 4 rocksdb: Options.inplace_update_support: 0 -1059> 2019-03-02 18:53:17.981942 7fe3640c4ec0 4 rocksdb: Options.inplace_update_num_locks: 10000 -1058> 2019-03-02 18:53:17.981943 7fe3640c4ec0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -1057> 2019-03-02 18:53:17.981944 7fe3640c4ec0 4 rocksdb: Options.memtable_huge_page_size: 0 -1056> 2019-03-02 18:53:17.981945 7fe3640c4ec0 4 rocksdb: Options.bloom_locality: 0 -1055> 2019-03-02 18:53:17.981948 7fe3640c4ec0 4 rocksdb: Options.max_successive_merges: 0 -1054> 2019-03-02 18:53:17.981949 7fe3640c4ec0 4 rocksdb: Options.optimize_filters_for_hits: 0 -1053> 2019-03-02 18:53:17.981950 7fe3640c4ec0 4 rocksdb: Options.paranoid_file_checks: 0 -1052> 2019-03-02 18:53:17.981950 7fe3640c4ec0 4 rocksdb: Options.force_consistency_checks: 0 -1051> 2019-03-02 18:53:17.981951 7fe3640c4ec0 4 rocksdb: Options.report_bg_io_stats: 0 -1050> 2019-03-02 18:53:17.982559 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-storage01roc/store.db/MANIFEST-000259 succeeded,manifest_file_number is 259, next_file_number is 261, last_sequence is 15983, log_number is 0,prev_log_number is 0,max_column_family is 0 -1049> 2019-03-02 18:53:17.982567 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 258 -1048> 2019-03-02 18:53:17.982631 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997982622, "job": 1, "event": "recovery_started", "log_files": [260]} -1047> 2019-03-02 18:53:17.982637 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:482] Recovering log #260 mode 2 -1046> 2019-03-02 18:53:17.983025 7fe3640c4ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:815] [default] [WriteLevel0TableForRecovery] Level-0 table #261: started -1045> 2019-03-02 18:53:17.983732 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997983708, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 261, "file_size": 50188, "table_properties": {"data_size": 48886, "index_size": 209, "filter_size": 176, "raw_key_size": 492, "raw_average_key_size": 24, "raw_value_size": 48300, "raw_average_value_size": 2415, "num_data_blocks": 7, "num_entries": 20, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} -1044> 2019-03-02 18:53:17.983745 7fe3640c4ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:847] [default] [WriteLevel0TableForRecovery] Level-0 table #261: 50188 bytes OK -1043> 2019-03-02 18:53:17.983765 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/version_set.cc:2395] Creating manifest 262 -1042> 2019-03-02 18:53:17.984320 7fe3640c4ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997984316, "job": 1, "event": "recovery_finished"} -1041> 2019-03-02 18:53:17.984519 7fe3640c4ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db//MANIFEST-000259 type=3 #259 -- OK -1040> 2019-03-02 18:53:17.984561 7fe3640c4ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db//000260.log type=0 #260 -- OK -1039> 2019-03-02 18:53:17.985556 7fe3640c4ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x560105348000 -1038> 2019-03-02 18:53:17.985720 7fe355efa700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- -1037> 2019-03-02 18:53:17.985728 7fe355efa700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl.cc:448] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 428.46 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 L1 1/0 26.55 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 5/0 26.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 12.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 12.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 428.46 KB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 L1 1/0 26.55 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 5/0 26.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.7 0 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 12.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** -1036> 2019-03-02 18:53:17.985835 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 4@0 + 1@1 files to L1, score 1.00 -1035> 2019-03-02 18:53:17.985852 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 0, inputs: [261(49KB) 258(270KB) 255(56KB) 252(53KB)], [250(26MB)] -1034> 2019-03-02 18:53:17.985879 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523997985856, "job": 3, "event": "compaction_started", "files_L0": [261, 258, 255, 252], "files_L1": [250], "score": 1, "input_data_size": 28275168} -1033> 2019-03-02 18:53:17.986482 7fe3576fd700 2 Event(0x560105253480 nevent=5000 time_id=1).set_owner idx=1 owner=140614401251072 -1032> 2019-03-02 18:53:17.986472 7fe356efc700 2 Event(0x560105258080 nevent=5000 time_id=1).set_owner idx=0 owner=140614392858368 -1031> 2019-03-02 18:53:17.986516 7fe35d709700 2 Event(0x560105253680 nevent=5000 time_id=1).set_owner idx=2 owner=140614501963520 -1030> 2019-03-02 18:53:17.986567 7fe3640c4ec0 0 starting mon.storage01roc rank 3 at public addr 192.168.253.203:6789/0 at bind addr 192.168.253.203:6789/0 mon_data /var/lib/ceph/mon/ceph-storage01roc fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea -1029> 2019-03-02 18:53:17.986768 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 learned_addr learned my addr 192.168.253.203:6789/0 -1028> 2019-03-02 18:53:17.986776 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _finish_bind bind my_inst.addr is 192.168.253.203:6789/0 -1027> 2019-03-02 18:53:17.986792 7fe3640c4ec0 0 starting mon.storage01roc rank 3 at 192.168.253.203:6789/0 mon_data /var/lib/ceph/mon/ceph-storage01roc fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea -1026> 2019-03-02 18:53:17.987367 7fe3640c4ec0 1 mon.storage01roc@-1(probing) e14 preinit fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea -1025> 2019-03-02 18:53:17.987560 7fe3640c4ec0 1 mon.storage01roc@-1(probing).mds e0 Unable to load 'last_metadata' -1024> 2019-03-02 18:53:17.987643 7fe3640c4ec0 1 mon.storage01roc@-1(probing).pg v0 on_upgrade discarding in-core PGMap -1023> 2019-03-02 18:53:17.987748 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mds e1 new map -1022> 2019-03-02 18:53:17.987752 7fe3640c4ec0 0 mon.storage01roc@-1(probing).mds e1 print_map e1 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2} legacy client fscid: -1 No filesystems configured -1021> 2019-03-02 18:53:17.988200 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires -1020> 2019-03-02 18:53:17.988209 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires -1019> 2019-03-02 18:53:17.988213 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 1009089991638532096, adjusting msgr requires -1018> 2019-03-02 18:53:17.988216 7fe3640c4ec0 0 mon.storage01roc@-1(probing).osd e79198 crush map has features 288514051259236352, adjusting msgr requires -1017> 2019-03-02 18:53:17.990514 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e0 loading version 519 -1016> 2019-03-02 18:53:17.990564 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e519 active server: 192.168.253.163:6820/2891295(9267543) -1015> 2019-03-02 18:53:17.990592 7fe3640c4ec0 4 mon.storage01roc@-1(probing).mgr e519 mkfs or daemon transitioned to available, loading commands -1014> 2019-03-02 18:53:17.990671 7fe3640c4ec0 4 mgrc handle_mgr_map Got map version 519 -1013> 2019-03-02 18:53:17.990679 7fe3640c4ec0 4 mgrc handle_mgr_map Active mgr is now 192.168.253.163:6820/2891295 -1012> 2019-03-02 18:53:17.990686 7fe3640c4ec0 4 mgrc reconnect Starting new session with 192.168.253.163:6820/2891295 -1011> 2019-03-02 18:53:17.990711 7fe3640c4ec0 1 -- - --> 192.168.253.163:6820/2891295 -- mgropen(unknown.storage01roc) v2 -- 0x560104faed80 con 0 -1010> 2019-03-02 18:53:17.991047 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 start start -1009> 2019-03-02 18:53:17.991056 7fe3640c4ec0 1 -- - start start -1008> 2019-03-02 18:53:17.991059 7fe3640c4ec0 2 mon.storage01roc@-1(probing) e14 init -1007> 2019-03-02 18:53:17.991175 7fe3640c4ec0 1 Processor -- start -1006> 2019-03-02 18:53:17.991261 7fe3640c4ec0 1 Processor -- start -1005> 2019-03-02 18:53:17.991316 7fe3640c4ec0 0 mon.storage01roc@-1(probing) e14 my rank is now 3 (was -1) -1004> 2019-03-02 18:53:17.991322 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 shutdown_connections -1003> 2019-03-02 18:53:17.991345 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6780 -1002> 2019-03-02 18:53:17.991363 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6780 con 0 -1001> 2019-03-02 18:53:17.991371 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6a00 -1000> 2019-03-02 18:53:17.991389 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6a00 con 0 -999> 2019-03-02 18:53:17.991394 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6c80 -998> 2019-03-02 18:53:17.991403 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6c80 con 0 -997> 2019-03-02 18:53:17.991407 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a6f00 -996> 2019-03-02 18:53:17.991418 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a6f00 con 0 -995> 2019-03-02 18:53:17.991422 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a7180 -994> 2019-03-02 18:53:17.991429 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a7180 con 0 -993> 2019-03-02 18:53:17.991433 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- ?+0 0x5601058a7400 -992> 2019-03-02 18:53:17.991458 7fe3640c4ec0 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage01roc) v6 -- 0x5601058a7400 con 0 -991> 2019-03-02 18:53:17.991794 7fe35d709700 1 -- 192.168.253.203:0/3016783 learned_addr learned my addr 192.168.253.203:0/3016783 -990> 2019-03-02 18:53:17.991868 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2577 chose new 2578 -989> 2019-03-02 18:53:17.991866 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -988> 2019-03-02 18:53:17.991889 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -987> 2019-03-02 18:53:17.991918 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -986> 2019-03-02 18:53:17.991931 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -985> 2019-03-02 18:53:17.991939 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2576 chose new 2579 -984> 2019-03-02 18:53:17.991962 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -983> 2019-03-02 18:53:17.991970 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -982> 2019-03-02 18:53:17.992394 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2577 chose new 2580 -981> 2019-03-02 18:53:17.992455 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 2576 chose new 2581 -980> 2019-03-02 18:53:17.992473 7fe35d709700 2 -- 192.168.253.203:0/3016783 >> 192.168.253.163:6820/2891295 conn(0x56010566f800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -979> 2019-03-02 18:53:17.992706 7fe3576fd700 0 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0).process missed message? skipped from seq 0 to 1798943791 -978> 2019-03-02 18:53:17.992727 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943791 0x5601058a7180 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage03roc paxos( fc 13804454 lc 13804982 )) v6 -977> 2019-03-02 18:53:17.992761 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943791 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage03roc paxos( fc 13804454 lc 13804982 )) v6 ==== 1014+0+0 (1244369052 0 0) 0x5601058a7180 con 0x560105678800 -976> 2019-03-02 18:53:17.992781 7fe356efc700 0 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0).process missed message? skipped from seq 0 to 1756793702 -975> 2019-03-02 18:53:17.992795 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection -974> 2019-03-02 18:53:17.992795 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793702 0x5601058a6f00 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage02roc paxos( fc 13804454 lc 13804982 )) v6 -973> 2019-03-02 18:53:17.992837 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793702 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name storage02roc paxos( fc 13804454 lc 13804982 )) v6 ==== 1014+0+0 (4205518415 0 0) 0x5601058a6f00 con 0x560105677000 -972> 2019-03-02 18:53:17.992848 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection -971> 2019-03-02 18:53:17.993079 7fe35d709700 5 -- 192.168.253.203:0/3016783 >> 192.168.253.163:6820/2891295 conn(0x56010566f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=8843 cs=1 l=1). rx mgr.9267543 seq 1 0x560105364f00 mgrconfigure(period=5, threshold=5) v2 -970> 2019-03-02 18:53:17.993106 7fe3586ff700 1 -- 192.168.253.203:0/3016783 <== mgr.9267543 192.168.253.163:6820/2891295 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x560105364f00 con 0x56010566f800 -969> 2019-03-02 18:53:17.993124 7fe3586ff700 4 mgrc handle_mgr_configure stats_period=5 -968> 2019-03-02 18:53:17.993126 7fe3586ff700 4 mgrc handle_mgr_configure updated stats threshold: 5 -967> 2019-03-02 18:53:17.993210 7fe3586ff700 1 -- 192.168.253.203:0/3016783 --> 192.168.253.163:6820/2891295 -- mgrreport(unknown.storage01roc +53-0 packed 798) v5 -- 0x5601056b0000 con 0 -966> 2019-03-02 18:53:17.993480 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 3480 chose new 3481 -965> 2019-03-02 18:53:17.994496 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got RETRY_GLOBAL 3480 chose new 3482 -964> 2019-03-02 18:53:17.995791 7fe35d709700 0 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0).process missed message? skipped from seq 0 to 1673216108 -963> 2019-03-02 18:53:17.995803 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216108 0x5601058a7400 mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon paxos( fc 13804454 lc 13804982 )) v6 -962> 2019-03-02 18:53:17.995816 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216108 ==== mon_probe(reply 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon paxos( fc 13804454 lc 13804982 )) v6 ==== 1009+0+0 (1769800333 0 0) 0x5601058a7400 con 0x56010567e000 -961> 2019-03-02 18:53:17.995827 7fe359f02700 5 mon.storage01roc@3(probing) e14 _ms_dispatch setting monitor caps on this connection -960> 2019-03-02 18:53:17.995856 7fe359f02700 0 log_channel(cluster) log [INF] : mon.storage01roc calling monitor election -959> 2019-03-02 18:53:17.995862 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 1 at 2019-03-02 18:53:17.995859) v1 -- 0x5601054f0d80 con 0 -958> 2019-03-02 18:53:17.995879 7fe359f02700 5 mon.storage01roc@3(electing).elector(0) start -- can i be leader? -957> 2019-03-02 18:53:17.995896 7fe359f02700 1 mon.storage01roc@3(electing).elector(58452) init, last seen epoch 58452 -956> 2019-03-02 18:53:17.996385 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500a300 -955> 2019-03-02 18:53:17.996394 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500a300 con 0 -954> 2019-03-02 18:53:17.996411 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500a900 -953> 2019-03-02 18:53:17.996415 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500a900 con 0 -952> 2019-03-02 18:53:17.996433 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500ac00 -951> 2019-03-02 18:53:17.996439 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500ac00 con 0 -950> 2019-03-02 18:53:17.996453 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500af00 -949> 2019-03-02 18:53:17.996458 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500af00 con 0 -948> 2019-03-02 18:53:17.996478 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500b200 -947> 2019-03-02 18:53:17.996483 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500b200 con 0 -946> 2019-03-02 18:53:17.996495 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- ?+0 0x56010500b500 -945> 2019-03-02 18:53:17.996500 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58453) v7 -- 0x56010500b500 con 0 -944> 2019-03-02 18:53:17.996514 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 1 at 2019-03-02 18:53:17.995859) v1 ==== 0+0+0 (0 0 0) 0x5601054f0d80 con 0x5601053ea000 -943> 2019-03-02 18:53:17.996526 7fe359f02700 5 mon.storage01roc@3(electing) e14 _ms_dispatch setting monitor caps on this connection -942> 2019-03-02 18:53:18.046452 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #266: 3947 keys, 28258990 bytes -941> 2019-03-02 18:53:18.046489 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998046470, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 266, "file_size": 28258990, "table_properties": {"data_size": 28101759, "index_size": 79039, "filter_size": 77265, "raw_key_size": 83782, "raw_average_key_size": 21, "raw_value_size": 27989473, "raw_average_value_size": 7091, "num_data_blocks": 2186, "num_entries": 3947, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} -940> 2019-03-02 18:53:18.046530 7fe355efa700 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 4@0 + 1@1 files to L1 => 28258990 bytes -939> 2019-03-02 18:53:18.047088 7fe355efa700 4 rocksdb: (Original Log Time 2019/03/02-18:53:18.046986) [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.11, MB/sec: 466.3 rd, 466.1 wr, level 1, files in(4, 1) out(1) MB in(0.4, 26.5) out(26.9), read-write-amplify(128.9) write-amplify(64.4) OK, records in: 3 -938> 2019-03-02 18:53:18.047100 7fe355efa700 4 rocksdb: (Original Log Time 2019/03/02-18:53:18.047014) EVENT_LOG_v1 {"time_micros": 1551523998047001, "job": 3, "event": "compaction_finished", "compaction_time_micros": 60635, "output_level": 1, "num_output_files": 1, "total_output_size": 28258990, "num_input_records": 3986, "num_output_records": 3947, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} -937> 2019-03-02 18:53:18.047190 7fe355efa700 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 3] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db/000261.sst type=2 #261 -- OK -936> 2019-03-02 18:53:18.047212 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047206, "job": 3, "event": "table_file_deletion", "file_number": 261} -935> 2019-03-02 18:53:18.047359 7fe355efa700 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 3] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db/000258.sst type=2 #258 -- OK -934> 2019-03-02 18:53:18.047375 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047371, "job": 3, "event": "table_file_deletion", "file_number": 258} -933> 2019-03-02 18:53:18.047443 7fe355efa700 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 3] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db/000255.sst type=2 #255 -- OK -932> 2019-03-02 18:53:18.047456 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047452, "job": 3, "event": "table_file_deletion", "file_number": 255} -931> 2019-03-02 18:53:18.047520 7fe355efa700 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 3] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db/000252.sst type=2 #252 -- OK -930> 2019-03-02 18:53:18.047533 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998047529, "job": 3, "event": "table_file_deletion", "file_number": 252} -929> 2019-03-02 18:53:18.055007 7fe355efa700 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5-2-ge988fb6/rpm/el7/BUILD/ceph-12.2.5-2-ge988fb6/src/rocksdb/db/db_impl_files.cc:307] [JOB 3] Delete /var/lib/ceph/mon/ceph-storage01roc/store.db/000250.sst type=2 #250 -- OK -928> 2019-03-02 18:53:18.055024 7fe355efa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1551523998055019, "job": 3, "event": "table_file_deletion", "file_number": 250} -927> 2019-03-02 18:53:18.113759 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=9 - -926> 2019-03-02 18:53:18.113873 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010543b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=10 - -925> 2019-03-02 18:53:18.113931 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2902 cs=1 l=1).handle_connect_msg accept write reply msg done -924> 2019-03-02 18:53:18.114022 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2902 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -923> 2019-03-02 18:53:18.114120 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=11 - -922> 2019-03-02 18:53:18.114141 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2902 cs=1 l=1). rx client.8113030 seq 1 0x5601054f06c0 auth(proto 0 34 bytes epoch 14) v1 -921> 2019-03-02 18:53:18.114174 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113030 192.168.253.32:0/3724611330 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2331791968 0 0) 0x5601054f06c0 con 0x56010549f000 -920> 2019-03-02 18:53:18.114206 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -919> 2019-03-02 18:53:18.114261 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13218 cs=1 l=1).handle_connect_msg accept write reply msg done -918> 2019-03-02 18:53:18.114326 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13218 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -917> 2019-03-02 18:53:18.114375 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010549c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=12 - -916> 2019-03-02 18:53:18.114384 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13218 cs=1 l=1). rx client.8831771 seq 1 0x5601054f0480 auth(proto 0 38 bytes epoch 14) v1 -915> 2019-03-02 18:53:18.114405 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8831771 192.168.253.43:0/103156267 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3029943862 0 0) 0x5601054f0480 con 0x56010549d800 -914> 2019-03-02 18:53:18.114421 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -913> 2019-03-02 18:53:18.114456 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279117 cs=1 l=1).handle_connect_msg accept write reply msg done -912> 2019-03-02 18:53:18.114924 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279117 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -911> 2019-03-02 18:53:18.114961 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279117 cs=1 l=1). rx osd.40 seq 1 0x560104faed80 auth(proto 0 27 bytes epoch 14) v1 -910> 2019-03-02 18:53:18.114988 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356684 cs=1 l=1).handle_connect_msg accept write reply msg done -909> 2019-03-02 18:53:18.114983 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3408158408 0 0) 0x560104faed80 con 0x56010543b800 -908> 2019-03-02 18:53:18.115016 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -907> 2019-03-02 18:53:18.115459 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b0000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=13 - -906> 2019-03-02 18:53:18.115512 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356684 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -905> 2019-03-02 18:53:18.115540 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356684 cs=1 l=1). rx osd.9 seq 1 0x5601054f1200 auth(proto 0 26 bytes epoch 14) v1 -904> 2019-03-02 18:53:18.115564 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (32594381 0 0) 0x5601054f1200 con 0x56010549c000 -903> 2019-03-02 18:53:18.115584 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -902> 2019-03-02 18:53:18.115957 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b1800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=28 - -901> 2019-03-02 18:53:18.115975 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b3000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - -900> 2019-03-02 18:53:18.115976 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601054b4800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=30 - -899> 2019-03-02 18:53:18.116078 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2872 cs=1 l=1).handle_connect_msg accept write reply msg done -898> 2019-03-02 18:53:18.116508 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35503 cs=1 l=1).handle_connect_msg accept write reply msg done -897> 2019-03-02 18:53:18.116523 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7242 cs=1 l=1).handle_connect_msg accept write reply msg done -896> 2019-03-02 18:53:18.116528 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18718 cs=1 l=1).handle_connect_msg accept write reply msg done -895> 2019-03-02 18:53:18.116613 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2872 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -894> 2019-03-02 18:53:18.116741 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2872 cs=1 l=1). rx client.8775011 seq 1 0x5601054f0900 auth(proto 0 34 bytes epoch 14) v1 -893> 2019-03-02 18:53:18.116768 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775011 192.168.253.39:0/1520712163 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2838151096 0 0) 0x5601054f0900 con 0x5601054b0000 -892> 2019-03-02 18:53:18.116785 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -891> 2019-03-02 18:53:18.117020 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18718 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -890> 2019-03-02 18:53:18.117080 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18718 cs=1 l=1). rx client.8833367 seq 1 0x5601054f1680 auth(proto 0 38 bytes epoch 14) v1 -889> 2019-03-02 18:53:18.117088 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7242 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -888> 2019-03-02 18:53:18.117088 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35503 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -887> 2019-03-02 18:53:18.117096 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8833367 192.168.253.50:0/3448699030 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (480927092 0 0) 0x5601054f1680 con 0x5601054b4800 -886> 2019-03-02 18:53:18.117108 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -885> 2019-03-02 18:53:18.117167 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35503 cs=1 l=1). rx client.8707891 seq 1 0x5601054f18c0 auth(proto 0 38 bytes epoch 14) v1 -884> 2019-03-02 18:53:18.117176 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7242 cs=1 l=1). rx client.8707856 seq 1 0x5601054f1d40 auth(proto 0 38 bytes epoch 14) v1 -883> 2019-03-02 18:53:18.117185 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8707891 192.168.253.31:0/1087877254 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1012484761 0 0) 0x5601054f18c0 con 0x5601054b1800 -882> 2019-03-02 18:53:18.117191 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -881> 2019-03-02 18:53:18.117194 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8707856 192.168.253.31:0/1038212884 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1570277103 0 0) 0x5601054f1d40 con 0x5601054b3000 -880> 2019-03-02 18:53:18.117199 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -879> 2019-03-02 18:53:18.177377 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=42 - -878> 2019-03-02 18:53:18.177578 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=14 cs=1 l=1).handle_connect_msg accept write reply msg done -877> 2019-03-02 18:53:18.177679 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=14 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -876> 2019-03-02 18:53:18.177855 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx client.? seq 1 0x560105492000 auth(proto 0 34 bytes epoch 0) v1 -875> 2019-03-02 18:53:18.177877 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.35:0/3362670835 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105492000 con 0x56010548c000 -874> 2019-03-02 18:53:18.177894 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 -873> 2019-03-02 18:53:18.191825 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -872> 2019-03-02 18:53:18.191829 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -871> 2019-03-02 18:53:18.191827 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -870> 2019-03-02 18:53:18.191838 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -869> 2019-03-02 18:53:18.191840 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -868> 2019-03-02 18:53:18.191841 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -867> 2019-03-02 18:53:18.341137 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=43 - -866> 2019-03-02 18:53:18.341703 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6 cs=1 l=1).handle_connect_msg accept write reply msg done -865> 2019-03-02 18:53:18.342271 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -864> 2019-03-02 18:53:18.342333 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6 cs=1 l=1). rx client.? seq 1 0x5601054f1b00 auth(proto 0 34 bytes epoch 0) v1 -863> 2019-03-02 18:53:18.342357 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.31:0/2755183883 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x5601054f1b00 con 0x56010548d800 -862> 2019-03-02 18:53:18.342379 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 -861> 2019-03-02 18:53:18.429977 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=44 - -860> 2019-03-02 18:53:18.429982 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105490800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=45 - -859> 2019-03-02 18:53:18.430248 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2956 cs=1 l=1).handle_connect_msg accept write reply msg done -858> 2019-03-02 18:53:18.430286 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3130 cs=1 l=1).handle_connect_msg accept write reply msg done -857> 2019-03-02 18:53:18.430372 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2956 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -856> 2019-03-02 18:53:18.430375 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3130 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -855> 2019-03-02 18:53:18.430465 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 1 0x5601054926c0 auth(proto 0 34 bytes epoch 14) v1 -854> 2019-03-02 18:53:18.430482 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3120407515 0 0) 0x5601054926c0 con 0x560105490800 -853> 2019-03-02 18:53:18.430504 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -852> 2019-03-02 18:53:18.430511 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 1 0x560105492d80 auth(proto 0 34 bytes epoch 14) v1 -851> 2019-03-02 18:53:18.430531 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (1595624629 0 0) 0x560105492d80 con 0x56010548f000 -850> 2019-03-02 18:53:18.430538 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -849> 2019-03-02 18:53:18.591771 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -848> 2019-03-02 18:53:18.591772 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -847> 2019-03-02 18:53:18.591790 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -846> 2019-03-02 18:53:18.591792 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -845> 2019-03-02 18:53:18.592768 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -844> 2019-03-02 18:53:18.592780 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -843> 2019-03-02 18:53:18.769480 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105638000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=46 - -842> 2019-03-02 18:53:18.769568 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135784 cs=1 l=1).handle_connect_msg accept write reply msg done -841> 2019-03-02 18:53:18.769580 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135784 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -840> 2019-03-02 18:53:18.769617 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135784 cs=1 l=1). rx osd.35 seq 1 0x560105492fc0 auth(proto 0 27 bytes epoch 14) v1 -839> 2019-03-02 18:53:18.769640 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3015307366 0 0) 0x560105492fc0 con 0x560105638000 -838> 2019-03-02 18:53:18.769662 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -837> 2019-03-02 18:53:18.769663 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105639800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=47 - -836> 2019-03-02 18:53:18.769782 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=86317 cs=1 l=1).handle_connect_msg accept write reply msg done -835> 2019-03-02 18:53:18.769829 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=86317 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -834> 2019-03-02 18:53:18.769864 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=86317 cs=1 l=1). rx osd.18 seq 1 0x560105492240 auth(proto 0 27 bytes epoch 14) v1 -833> 2019-03-02 18:53:18.769878 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.18 192.168.253.205:6816/641907 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3382322512 0 0) 0x560105492240 con 0x560105639800 -832> 2019-03-02 18:53:18.769885 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -831> 2019-03-02 18:53:18.770288 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=48 - -830> 2019-03-02 18:53:18.770432 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=49 - -829> 2019-03-02 18:53:18.770832 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=465203 cs=1 l=1).handle_connect_msg accept write reply msg done -828> 2019-03-02 18:53:18.771033 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3774 cs=1 l=1).handle_connect_msg accept write reply msg done -827> 2019-03-02 18:53:18.771300 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=465203 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -826> 2019-03-02 18:53:18.771340 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 1 0x560105492900 auth(proto 0 27 bytes epoch 14) v1 -825> 2019-03-02 18:53:18.771356 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3123283281 0 0) 0x560105492900 con 0x56010563b000 -824> 2019-03-02 18:53:18.771365 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -823> 2019-03-02 18:53:18.771496 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3774 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -822> 2019-03-02 18:53:18.771534 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3774 cs=1 l=1). rx client.9267543 seq 1 0x560105493200 auth(proto 0 37 bytes epoch 14) v1 -821> 2019-03-02 18:53:18.771549 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 1 ==== auth(proto 0 37 bytes epoch 14) v1 ==== 67+0+0 (3305926266 0 0) 0x560105493200 con 0x56010563c800 -820> 2019-03-02 18:53:18.771557 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 37 bytes epoch 14) v1 -819> 2019-03-02 18:53:18.785093 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545a000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=50 - -818> 2019-03-02 18:53:18.785186 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=95151 cs=1 l=1).handle_connect_msg accept write reply msg done -817> 2019-03-02 18:53:18.785233 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=95151 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -816> 2019-03-02 18:53:18.785268 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 1 0x560105492480 auth(proto 0 26 bytes epoch 14) v1 -815> 2019-03-02 18:53:18.785294 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2633358351 0 0) 0x560105492480 con 0x56010545a000 -814> 2019-03-02 18:53:18.785310 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -813> 2019-03-02 18:53:19.132954 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=51 - -812> 2019-03-02 18:53:19.132999 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545d000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=52 - -811> 2019-03-02 18:53:19.133056 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=96529 cs=1 l=1).handle_connect_msg accept write reply msg done -810> 2019-03-02 18:53:19.133097 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=96529 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -809> 2019-03-02 18:53:19.133130 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=96529 cs=1 l=1). rx osd.12 seq 1 0x560105492b40 auth(proto 0 27 bytes epoch 14) v1 -808> 2019-03-02 18:53:19.133154 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.12 192.168.253.205:6814/641212 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (2274412844 0 0) 0x560105492b40 con 0x56010545b800 -807> 2019-03-02 18:53:19.133172 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -806> 2019-03-02 18:53:19.133305 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545e800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=53 - -805> 2019-03-02 18:53:19.133498 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f4000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=54 - -804> 2019-03-02 18:53:19.133521 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=353413 cs=1 l=1).handle_connect_msg accept write reply msg done -803> 2019-03-02 18:53:19.133639 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2943 cs=1 l=1).handle_connect_msg accept write reply msg done -802> 2019-03-02 18:53:19.133696 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2943 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -801> 2019-03-02 18:53:19.133747 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2943 cs=1 l=1). rx client.8775228 seq 1 0x560105493440 auth(proto 0 34 bytes epoch 14) v1 -800> 2019-03-02 18:53:19.133760 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f5800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=55 - -799> 2019-03-02 18:53:19.133767 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f7000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=56 - -798> 2019-03-02 18:53:19.133764 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775228 192.168.253.32:0/360631102 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (1894617780 0 0) 0x560105493440 con 0x5601056f4000 -797> 2019-03-02 18:53:19.133800 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601056f8800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=57 - -796> 2019-03-02 18:53:19.133803 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -795> 2019-03-02 18:53:19.133835 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=504598 cs=1 l=1).handle_connect_msg accept write reply msg done -794> 2019-03-02 18:53:19.133853 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105758000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=58 - -793> 2019-03-02 18:53:19.133927 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=49224 cs=1 l=1).handle_connect_msg accept write reply msg done -792> 2019-03-02 18:53:19.133966 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7222 cs=1 l=1).handle_connect_msg accept write reply msg done -791> 2019-03-02 18:53:19.133970 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=353413 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -790> 2019-03-02 18:53:19.133999 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=353413 cs=1 l=1). rx osd.0 seq 1 0x560105493d40 auth(proto 0 26 bytes epoch 14) v1 -789> 2019-03-02 18:53:19.134025 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=49224 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -788> 2019-03-02 18:53:19.134022 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.0 192.168.253.164:6806/342760 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2286119786 0 0) 0x560105493d40 con 0x56010545d000 -787> 2019-03-02 18:53:19.134038 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -786> 2019-03-02 18:53:19.134047 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=7222 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -785> 2019-03-02 18:53:19.134085 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=49224 cs=1 l=1). rx client.8947140 seq 1 0x560105434000 auth(proto 0 38 bytes epoch 14) v1 -784> 2019-03-02 18:53:19.134101 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8947140 192.168.253.46:0/836723901 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3760714466 0 0) 0x560105434000 con 0x5601056f5800 -783> 2019-03-02 18:53:19.134107 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 1 0x560105434b40 auth(proto 0 34 bytes epoch 14) v1 -782> 2019-03-02 18:53:19.134117 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -781> 2019-03-02 18:53:19.134120 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3816987468 0 0) 0x560105434b40 con 0x560105758000 -780> 2019-03-02 18:53:19.134127 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -779> 2019-03-02 18:53:19.134254 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105759800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=59 - -778> 2019-03-02 18:53:19.134305 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=504598 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -777> 2019-03-02 18:53:19.134315 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=411209 cs=1 l=1).handle_connect_msg accept write reply msg done -776> 2019-03-02 18:53:19.134342 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=504598 cs=1 l=1). rx osd.8 seq 1 0x560105434d80 auth(proto 0 26 bytes epoch 14) v1 -775> 2019-03-02 18:53:19.134356 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.8 192.168.253.163:6812/345669 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (1449422260 0 0) 0x560105434d80 con 0x56010545e800 -774> 2019-03-02 18:53:19.134373 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -773> 2019-03-02 18:53:19.134397 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2894 cs=1 l=1).handle_connect_msg accept write reply msg done -772> 2019-03-02 18:53:19.134429 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6880 cs=1 l=1).handle_connect_msg accept write reply msg done -771> 2019-03-02 18:53:19.134495 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6880 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -770> 2019-03-02 18:53:19.134571 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6880 cs=1 l=1). rx client.8945978 seq 1 0x560105493680 auth(proto 0 38 bytes epoch 14) v1 -769> 2019-03-02 18:53:19.134586 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8945978 192.168.253.46:0/2587405440 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (434972468 0 0) 0x560105493680 con 0x560105759800 -768> 2019-03-02 18:53:19.134600 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -767> 2019-03-02 18:53:19.134803 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=411209 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -766> 2019-03-02 18:53:19.134840 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=411209 cs=1 l=1). rx osd.45 seq 1 0x560105434240 auth(proto 0 27 bytes epoch 14) v1 -765> 2019-03-02 18:53:19.134854 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.45 192.168.253.165:6812/710509 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (2375892178 0 0) 0x560105434240 con 0x5601056f8800 -764> 2019-03-02 18:53:19.134862 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -763> 2019-03-02 18:53:19.134919 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2894 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -762> 2019-03-02 18:53:19.135009 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2894 cs=1 l=1). rx client.8112890 seq 1 0x5601054938c0 auth(proto 0 34 bytes epoch 14) v1 -761> 2019-03-02 18:53:19.135023 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8112890 192.168.253.39:0/282811283 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (435497577 0 0) 0x5601054938c0 con 0x5601056f7000 -760> 2019-03-02 18:53:19.135028 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -759> 2019-03-02 18:53:19.292967 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010575b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=60 - -758> 2019-03-02 18:53:19.293498 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=189842 cs=1 l=1).handle_connect_msg accept write reply msg done -757> 2019-03-02 18:53:19.293957 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=189842 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -756> 2019-03-02 18:53:19.293985 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 1 0x560105434480 auth(proto 0 27 bytes epoch 14) v1 -755> 2019-03-02 18:53:19.293999 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3917320626 0 0) 0x560105434480 con 0x56010575b000 -754> 2019-03-02 18:53:19.294016 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -753> 2019-03-02 18:53:19.391772 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -752> 2019-03-02 18:53:19.391785 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -751> 2019-03-02 18:53:19.391785 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -750> 2019-03-02 18:53:19.391799 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -749> 2019-03-02 18:53:19.392773 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -748> 2019-03-02 18:53:19.392785 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -747> 2019-03-02 18:53:19.782889 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).read_bulk peer close file descriptor 51 -746> 2019-03-02 18:53:19.782904 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).read_until read failed -745> 2019-03-02 18:53:19.782916 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).process read tag failed -744> 2019-03-02 18:53:19.782923 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1).fault on lossy channel, failing -743> 2019-03-02 18:53:19.782928 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6814/641212 conn(0x56010545b800 :6789 s=STATE_OPEN pgs=96529 cs=1 l=1)._stop -742> 2019-03-02 18:53:19.861758 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).read_bulk peer close file descriptor 49 -741> 2019-03-02 18:53:19.861771 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).read_until read failed -740> 2019-03-02 18:53:19.861780 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).process read tag failed -739> 2019-03-02 18:53:19.861787 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1).fault on lossy channel, failing -738> 2019-03-02 18:53:19.861792 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=3774 cs=1 l=1)._stop -737> 2019-03-02 18:53:19.862298 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010575c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=49 - -736> 2019-03-02 18:53:19.862849 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3778 cs=1 l=1).handle_connect_msg accept write reply msg done -735> 2019-03-02 18:53:19.863299 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3778 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -734> 2019-03-02 18:53:19.863331 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 1 0x5601054346c0 auth(proto 0 37 bytes epoch 14) v1 -733> 2019-03-02 18:53:19.863352 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 1 ==== auth(proto 0 37 bytes epoch 14) v1 ==== 67+0+0 (3305926266 0 0) 0x5601054346c0 con 0x56010575c800 -732> 2019-03-02 18:53:19.863374 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 37 bytes epoch 14) v1 -731> 2019-03-02 18:53:19.891302 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).read_bulk peer close file descriptor 10 -730> 2019-03-02 18:53:19.891314 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).read_until read failed -729> 2019-03-02 18:53:19.891319 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).process read tag failed -728> 2019-03-02 18:53:19.891322 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1).fault on lossy channel, failing -727> 2019-03-02 18:53:19.891325 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x56010543b800 :6789 s=STATE_OPEN pgs=279117 cs=1 l=1)._stop -726> 2019-03-02 18:53:19.891847 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105772000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=10 - -725> 2019-03-02 18:53:19.892399 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279118 cs=1 l=1).handle_connect_msg accept write reply msg done -724> 2019-03-02 18:53:19.892857 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=279118 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -723> 2019-03-02 18:53:19.892885 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 1 0x560105434fc0 auth(proto 0 27 bytes epoch 14) v1 -722> 2019-03-02 18:53:19.892907 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3408158408 0 0) 0x560105434fc0 con 0x560105772000 -721> 2019-03-02 18:53:19.892926 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -720> 2019-03-02 18:53:20.187672 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105773800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=51 - -719> 2019-03-02 18:53:20.187740 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=89395 cs=1 l=1).handle_connect_msg accept write reply msg done -718> 2019-03-02 18:53:20.187749 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=89395 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -717> 2019-03-02 18:53:20.187790 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 1 0x560105434900 auth(proto 0 26 bytes epoch 14) v1 -716> 2019-03-02 18:53:20.187808 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2020154301 0 0) 0x560105434900 con 0x560105773800 -715> 2019-03-02 18:53:20.187826 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -714> 2019-03-02 18:53:20.527395 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105775000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=61 - -713> 2019-03-02 18:53:20.527559 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11207 cs=1 l=1).handle_connect_msg accept write reply msg done -712> 2019-03-02 18:53:20.527629 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11207 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -711> 2019-03-02 18:53:20.527710 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 1 0x560105435200 auth(proto 0 38 bytes epoch 14) v1 -710> 2019-03-02 18:53:20.527730 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (2078396002 0 0) 0x560105435200 con 0x560105775000 -709> 2019-03-02 18:53:20.527747 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -708> 2019-03-02 18:53:20.738669 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).read_bulk peer close file descriptor 43 -707> 2019-03-02 18:53:20.738683 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).read_until read failed -706> 2019-03-02 18:53:20.738694 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).process read tag failed -705> 2019-03-02 18:53:20.738701 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1).fault on lossy channel, failing -704> 2019-03-02 18:53:20.738707 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010548d800 :6789 s=STATE_OPEN pgs=6 cs=1 l=1)._stop -703> 2019-03-02 18:53:20.753597 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105776800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=43 - -702> 2019-03-02 18:53:20.753639 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105788000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=62 - -701> 2019-03-02 18:53:20.754182 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12571 cs=1 l=1).handle_connect_msg accept write reply msg done -700> 2019-03-02 18:53:20.754213 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=4540 cs=1 l=1).handle_connect_msg accept write reply msg done -699> 2019-03-02 18:53:20.754755 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12571 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -698> 2019-03-02 18:53:20.754777 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=4540 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -697> 2019-03-02 18:53:20.754825 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 1 0x560105493b00 auth(proto 0 38 bytes epoch 14) v1 -696> 2019-03-02 18:53:20.754839 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (4053673190 0 0) 0x560105493b00 con 0x560105776800 -695> 2019-03-02 18:53:20.754859 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -694> 2019-03-02 18:53:20.754860 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 1 0x560105435440 auth(proto 0 38 bytes epoch 14) v1 -693> 2019-03-02 18:53:20.754872 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (1470431156 0 0) 0x560105435440 con 0x560105788000 -692> 2019-03-02 18:53:20.754881 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -691> 2019-03-02 18:53:20.811243 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105789800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=63 - -690> 2019-03-02 18:53:20.811396 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6463 cs=1 l=1).handle_connect_msg accept write reply msg done -689> 2019-03-02 18:53:20.811494 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6463 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -688> 2019-03-02 18:53:20.811612 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 1 0x560105794000 auth(proto 0 38 bytes epoch 14) v1 -687> 2019-03-02 18:53:20.811633 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3722122544 0 0) 0x560105794000 con 0x560105789800 -686> 2019-03-02 18:53:20.811650 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -685> 2019-03-02 18:53:20.991777 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -684> 2019-03-02 18:53:20.991778 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -683> 2019-03-02 18:53:20.991792 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -682> 2019-03-02 18:53:20.991794 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -681> 2019-03-02 18:53:20.992771 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -680> 2019-03-02 18:53:20.992782 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -679> 2019-03-02 18:53:21.177128 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).read_bulk peer close file descriptor 42 -678> 2019-03-02 18:53:21.177141 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).read_until read failed -677> 2019-03-02 18:53:21.177145 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).process read tag failed -676> 2019-03-02 18:53:21.177149 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1).fault on lossy channel, failing -675> 2019-03-02 18:53:21.177152 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3362670835 conn(0x56010548c000 :6789 s=STATE_OPEN pgs=14 cs=1 l=1)._stop -674> 2019-03-02 18:53:21.192635 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010578b000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=42 - -673> 2019-03-02 18:53:21.193203 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=54339 cs=1 l=1).handle_connect_msg accept write reply msg done -672> 2019-03-02 18:53:21.193861 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=54339 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -671> 2019-03-02 18:53:21.194000 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 1 0x560105794d80 auth(proto 0 38 bytes epoch 14) v1 -670> 2019-03-02 18:53:21.194022 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.7893124 192.168.253.44:0/219447902 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3817785262 0 0) 0x560105794d80 con 0x56010578b000 -669> 2019-03-02 18:53:21.194041 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -668> 2019-03-02 18:53:21.281260 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010578c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=64 - -667> 2019-03-02 18:53:21.281445 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=28 cs=1 l=1).handle_connect_msg accept write reply msg done -666> 2019-03-02 18:53:21.281607 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=28 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -665> 2019-03-02 18:53:21.281697 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=28 cs=1 l=1). rx client.? seq 1 0x560105794240 auth(proto 0 34 bytes epoch 0) v1 -664> 2019-03-02 18:53:21.281719 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.46:0/2878661900 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105794240 con 0x56010578c800 -663> 2019-03-02 18:53:21.281737 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 -662> 2019-03-02 18:53:21.985027 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058f8000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=65 - -661> 2019-03-02 18:53:21.985125 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058f9800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=66 - -660> 2019-03-02 18:53:21.985617 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18715 cs=1 l=1).handle_connect_msg accept write reply msg done -659> 2019-03-02 18:53:21.985681 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13944 cs=1 l=1).handle_connect_msg accept write reply msg done -658> 2019-03-02 18:53:21.986105 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=18715 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -657> 2019-03-02 18:53:21.986193 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 1 0x560105794fc0 auth(proto 0 38 bytes epoch 14) v1 -656> 2019-03-02 18:53:21.986209 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (875330357 0 0) 0x560105794fc0 con 0x5601058f8000 -655> 2019-03-02 18:53:21.986228 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -654> 2019-03-02 18:53:21.986266 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=13944 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -653> 2019-03-02 18:53:21.986337 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 1 0x560105435680 auth(proto 0 38 bytes epoch 14) v1 -652> 2019-03-02 18:53:21.986354 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (2336192628 0 0) 0x560105435680 con 0x5601058f9800 -651> 2019-03-02 18:53:21.986367 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -650> 2019-03-02 18:53:22.141538 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793703 0x56010500af00 election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58459) v7 -649> 2019-03-02 18:53:22.141575 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793703 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58459) v7 ==== 1023+0+0 (831641765 0 0) 0x56010500af00 con 0x560105677000 -648> 2019-03-02 18:53:22.141623 7fe359f02700 5 mon.storage01roc@3(electing).elector(58453) handle_propose from mon.4 -647> 2019-03-02 18:53:22.141840 7fe359f02700 0 log_channel(cluster) log [INF] : mon.storage01roc calling monitor election -646> 2019-03-02 18:53:22.141849 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 2 at 2019-03-02 18:53:22.141845) v1 -- 0x5601057958c0 con 0 -645> 2019-03-02 18:53:22.141865 7fe359f02700 5 mon.storage01roc@3(electing).elector(58459) start -- can i be leader? -644> 2019-03-02 18:53:22.141875 7fe359f02700 1 mon.storage01roc@3(electing).elector(58459) init, last seen epoch 58459, mid-election, bumping -643> 2019-03-02 18:53:22.142435 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x56010500b800 -642> 2019-03-02 18:53:22.142447 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x56010500b800 con 0 -641> 2019-03-02 18:53:22.142464 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x56010500bb00 -640> 2019-03-02 18:53:22.142468 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x56010500bb00 con 0 -639> 2019-03-02 18:53:22.142499 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908000 -638> 2019-03-02 18:53:22.142511 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908000 con 0 -637> 2019-03-02 18:53:22.142535 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908300 -636> 2019-03-02 18:53:22.142539 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908300 con 0 -635> 2019-03-02 18:53:22.142552 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908600 -634> 2019-03-02 18:53:22.142561 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908600 con 0 -633> 2019-03-02 18:53:22.142580 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- ?+0 0x560105908900 -632> 2019-03-02 18:53:22.142588 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea propose 58461) v7 -- 0x560105908900 con 0 -631> 2019-03-02 18:53:22.142613 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 2 at 2019-03-02 18:53:22.141845) v1 ==== 0+0+0 (0 0 0) 0x5601057958c0 con 0x5601053ea000 -630> 2019-03-02 18:53:22.143244 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943792 0x560105908600 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 -629> 2019-03-02 18:53:22.143270 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943792 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40498+0+0 (3500033299 0 0) 0x560105908600 con 0x560105678800 -628> 2019-03-02 18:53:22.143300 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793704 0x560105908300 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 -627> 2019-03-02 18:53:22.143323 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.5 -626> 2019-03-02 18:53:22.143329 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]) } -625> 2019-03-02 18:53:22.143345 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793704 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40498+0+0 (2531979772 0 0) 0x560105908300 con 0x560105677000 -624> 2019-03-02 18:53:22.143368 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.4 -623> 2019-03-02 18:53:22.143373 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.4: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]) } -622> 2019-03-02 18:53:22.153619 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216109 0x560105908900 election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 -621> 2019-03-02 18:53:22.153647 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216109 ==== election(484eca8f-c17b-41fd-83eb-314052ccd8ea ack 58461) v7 ==== 40488+0+0 (2345732302 0 0) 0x560105908900 con 0x56010567e000 -620> 2019-03-02 18:53:22.153679 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) handle_ack from mon.6 -619> 2019-03-02 18:53:22.153686 7fe359f02700 5 mon.storage01roc@3(electing).elector(58461) so far i have { mon.3: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.4: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.5: features 2305244844532236283 mon_feature_t([kraken,luminous]), mon.6: features 2305244844532236283 mon_feature_t([kraken,luminous]) } -618> 2019-03-02 18:53:22.993342 7fe358f00700 1 -- 192.168.253.203:0/3016783 --> 192.168.253.163:6820/2891295 -- mgrreport(unknown.storage01roc +0-0 packed 798) v5 -- 0x5601056b02c0 con 0 -617> 2019-03-02 18:53:23.063662 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058fb000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=67 - -616> 2019-03-02 18:53:23.064262 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=5417 cs=1 l=1).handle_connect_msg accept write reply msg done -615> 2019-03-02 18:53:23.064813 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=5417 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -614> 2019-03-02 18:53:23.064945 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 1 0x560105794480 auth(proto 0 34 bytes epoch 14) v1 -613> 2019-03-02 18:53:23.064968 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2725449133 0 0) 0x560105794480 con 0x5601058fb000 -612> 2019-03-02 18:53:23.064988 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -611> 2019-03-02 18:53:23.086826 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x5601058fc800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=68 - -610> 2019-03-02 18:53:23.087017 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11821 cs=1 l=1).handle_connect_msg accept write reply msg done -609> 2019-03-02 18:53:23.087130 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=11821 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -608> 2019-03-02 18:53:23.087292 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 1 0x560105795200 auth(proto 0 38 bytes epoch 14) v1 -607> 2019-03-02 18:53:23.087313 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (98846487 0 0) 0x560105795200 con 0x5601058fc800 -606> 2019-03-02 18:53:23.087329 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -605> 2019-03-02 18:53:23.240286 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).read_bulk peer close file descriptor 29 -604> 2019-03-02 18:53:23.240299 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).read_until read failed -603> 2019-03-02 18:53:23.240303 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).process read tag failed -602> 2019-03-02 18:53:23.240309 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1).fault on lossy channel, failing -601> 2019-03-02 18:53:23.240312 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1038212884 conn(0x5601054b3000 :6789 s=STATE_OPEN pgs=7242 cs=1 l=1)._stop -600> 2019-03-02 18:53:23.740441 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010563c800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - -599> 2019-03-02 18:53:23.741038 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12 cs=1 l=1).handle_connect_msg accept write reply msg done -598> 2019-03-02 18:53:23.741515 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=12 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -597> 2019-03-02 18:53:23.741671 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx client.? seq 1 0x560105795440 auth(proto 0 34 bytes epoch 0) v1 -596> 2019-03-02 18:53:23.741689 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.31:0/2755183883 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x560105795440 con 0x56010563c800 -595> 2019-03-02 18:53:23.741706 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 -594> 2019-03-02 18:53:23.818121 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).read_bulk peer close file descriptor 11 -593> 2019-03-02 18:53:23.818134 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).read_until read failed -592> 2019-03-02 18:53:23.818137 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).process read tag failed -591> 2019-03-02 18:53:23.818142 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1).fault on lossy channel, failing -590> 2019-03-02 18:53:23.818145 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/103156267 conn(0x56010549d800 :6789 s=STATE_OPEN pgs=13218 cs=1 l=1)._stop -589> 2019-03-02 18:53:23.858950 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010543b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=11 - -588> 2019-03-02 18:53:23.858973 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010593e000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=69 - -587> 2019-03-02 18:53:23.859494 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6891 cs=1 l=1).handle_connect_msg accept write reply msg done -586> 2019-03-02 18:53:23.859531 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35760 cs=1 l=1).handle_connect_msg accept write reply msg done -585> 2019-03-02 18:53:23.860064 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=6891 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -584> 2019-03-02 18:53:23.860116 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=35760 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -583> 2019-03-02 18:53:23.860631 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 1 0x5601054358c0 auth(proto 0 38 bytes epoch 14) v1 -582> 2019-03-02 18:53:23.860636 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 1 0x560105795680 auth(proto 0 38 bytes epoch 14) v1 -581> 2019-03-02 18:53:23.860655 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (3214538620 0 0) 0x5601054358c0 con 0x56010543b800 -580> 2019-03-02 18:53:23.860675 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -579> 2019-03-02 18:53:23.860683 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 1 ==== auth(proto 0 38 bytes epoch 14) v1 ==== 68+0+0 (701291500 0 0) 0x560105795680 con 0x56010593e000 -578> 2019-03-02 18:53:23.860690 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 38 bytes epoch 14) v1 -577> 2019-03-02 18:53:24.153794 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).read_bulk peer close file descriptor 52 -576> 2019-03-02 18:53:24.153812 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).read_until read failed -575> 2019-03-02 18:53:24.153818 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).process read tag failed -574> 2019-03-02 18:53:24.153824 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1).fault on lossy channel, failing -573> 2019-03-02 18:53:24.153827 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6806/342760 conn(0x56010545d000 :6789 s=STATE_OPEN pgs=353413 cs=1 l=1)._stop -572> 2019-03-02 18:53:24.153837 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start -571> 2019-03-02 18:53:24.153839 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545b800 -570> 2019-03-02 18:53:24.153844 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545d000 -569> 2019-03-02 18:53:24.153846 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010548c000 -568> 2019-03-02 18:53:24.153852 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010549d800 -567> 2019-03-02 18:53:24.153856 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b3000 -566> 2019-03-02 18:53:24.191773 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -565> 2019-03-02 18:53:24.191775 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -564> 2019-03-02 18:53:24.191787 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6789/0 conn(0x560105672800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -563> 2019-03-02 18:53:24.191791 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6789/0 conn(0x560105674000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -562> 2019-03-02 18:53:24.192771 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed -561> 2019-03-02 18:53:24.192783 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6789/0 conn(0x560105675800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused! -560> 2019-03-02 18:53:24.281656 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).read_bulk peer close file descriptor 64 -559> 2019-03-02 18:53:24.281668 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).read_until read failed -558> 2019-03-02 18:53:24.281672 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).process read tag failed -557> 2019-03-02 18:53:24.281686 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1).fault on lossy channel, failing -556> 2019-03-02 18:53:24.281690 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2878661900 conn(0x56010578c800 :6789 s=STATE_OPEN pgs=28 cs=1 l=1)._stop -555> 2019-03-02 18:53:24.298590 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).read_bulk peer close file descriptor 13 -554> 2019-03-02 18:53:24.298603 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).read_until read failed -553> 2019-03-02 18:53:24.298607 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).process read tag failed -552> 2019-03-02 18:53:24.298610 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1).fault on lossy channel, failing -551> 2019-03-02 18:53:24.298613 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/1520712163 conn(0x5601054b0000 :6789 s=STATE_OPEN pgs=2872 cs=1 l=1)._stop -550> 2019-03-02 18:53:24.299896 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).read_bulk peer close file descriptor 56 -549> 2019-03-02 18:53:24.299908 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).read_until read failed -548> 2019-03-02 18:53:24.299912 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).process read tag failed -547> 2019-03-02 18:53:24.299916 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1).fault on lossy channel, failing -546> 2019-03-02 18:53:24.299919 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.39:0/282811283 conn(0x5601056f7000 :6789 s=STATE_OPEN pgs=2894 cs=1 l=1)._stop -545> 2019-03-02 18:53:24.557504 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010548c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=13 - -544> 2019-03-02 18:53:24.557696 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=1 cs=1 l=1).handle_connect_msg accept write reply msg done -543> 2019-03-02 18:53:24.557765 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=1 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -542> 2019-03-02 18:53:24.557881 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.52:0/3887599110 conn(0x56010548c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1 cs=1 l=1). rx client.? seq 1 0x5601057946c0 auth(proto 0 34 bytes epoch 0) v1 -541> 2019-03-02 18:53:24.557899 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.? 192.168.253.52:0/3887599110 1 ==== auth(proto 0 34 bytes epoch 0) v1 ==== 64+0+0 (3805625626 0 0) 0x5601057946c0 con 0x56010548c000 -540> 2019-03-02 18:53:24.557916 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 0) v1 -539> 2019-03-02 18:53:24.685869 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).read_bulk peer close file descriptor 53 -538> 2019-03-02 18:53:24.685883 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).read_until read failed -537> 2019-03-02 18:53:24.685886 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).process read tag failed -536> 2019-03-02 18:53:24.685890 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1).fault on lossy channel, failing -535> 2019-03-02 18:53:24.685893 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.163:6812/345669 conn(0x56010545e800 :6789 s=STATE_OPEN pgs=504598 cs=1 l=1)._stop -534> 2019-03-02 18:53:25.253219 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010545b800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=52 - -533> 2019-03-02 18:53:25.253831 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=330841 cs=1 l=1).handle_connect_msg accept write reply msg done -532> 2019-03-02 18:53:25.254298 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=330841 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -531> 2019-03-02 18:53:25.254349 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 1 0x560105493200 auth(proto 0 27 bytes epoch 14) v1 -530> 2019-03-02 18:53:25.254367 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (4213901812 0 0) 0x560105493200 con 0x56010545b800 -529> 2019-03-02 18:53:25.254384 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -528> 2019-03-02 18:53:25.293984 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).read_bulk peer close file descriptor 30 -527> 2019-03-02 18:53:25.293997 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).read_until read failed -526> 2019-03-02 18:53:25.294001 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).process read tag failed -525> 2019-03-02 18:53:25.294004 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1).fault on lossy channel, failing -524> 2019-03-02 18:53:25.294008 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.50:0/3448699030 conn(0x5601054b4800 :6789 s=STATE_OPEN pgs=18718 cs=1 l=1)._stop -523> 2019-03-02 18:53:25.294017 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start -522> 2019-03-02 18:53:25.294019 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010545e800 -521> 2019-03-02 18:53:25.294033 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b0000 -520> 2019-03-02 18:53:25.294037 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601054b4800 -519> 2019-03-02 18:53:25.294040 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f7000 -518> 2019-03-02 18:53:25.294043 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010578c800 -517> 2019-03-02 18:53:25.397219 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).read_bulk peer close file descriptor 59 -516> 2019-03-02 18:53:25.397232 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).read_until read failed -515> 2019-03-02 18:53:25.397236 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).process read tag failed -514> 2019-03-02 18:53:25.397240 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1).fault on lossy channel, failing -513> 2019-03-02 18:53:25.397243 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/2587405440 conn(0x560105759800 :6789 s=STATE_OPEN pgs=6880 cs=1 l=1)._stop -512> 2019-03-02 18:53:25.452621 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010593f800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=30 - -511> 2019-03-02 18:53:25.452801 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=68711 cs=1 l=1).handle_connect_msg accept write reply msg done -510> 2019-03-02 18:53:25.452839 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=68711 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -509> 2019-03-02 18:53:25.452903 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 1 0x560105794900 auth(proto 0 26 bytes epoch 14) v1 -508> 2019-03-02 18:53:25.452923 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (3612044724 0 0) 0x560105794900 con 0x56010593f800 -507> 2019-03-02 18:53:25.452942 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -506> 2019-03-02 18:53:25.624523 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).read_bulk peer close file descriptor 55 -505> 2019-03-02 18:53:25.624536 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).read_until read failed -504> 2019-03-02 18:53:25.624541 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).process read tag failed -503> 2019-03-02 18:53:25.624544 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1).fault on lossy channel, failing -502> 2019-03-02 18:53:25.624547 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.46:0/836723901 conn(0x5601056f5800 :6789 s=STATE_OPEN pgs=49224 cs=1 l=1)._stop -501> 2019-03-02 18:53:25.636883 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105941000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=53 - -500> 2019-03-02 18:53:25.637444 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=245753 cs=1 l=1).handle_connect_msg accept write reply msg done -499> 2019-03-02 18:53:25.637900 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=245753 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -498> 2019-03-02 18:53:25.637927 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 1 0x560105794b40 auth(proto 0 27 bytes epoch 14) v1 -497> 2019-03-02 18:53:25.637941 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3528568399 0 0) 0x560105794b40 con 0x560105941000 -496> 2019-03-02 18:53:25.637958 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -495> 2019-03-02 18:53:25.761886 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).read_bulk peer close file descriptor 12 -494> 2019-03-02 18:53:25.761899 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).read_until read failed -493> 2019-03-02 18:53:25.761903 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).process read tag failed -492> 2019-03-02 18:53:25.761907 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1).fault on lossy channel, failing -491> 2019-03-02 18:53:25.761910 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x56010549c000 :6789 s=STATE_OPEN pgs=356684 cs=1 l=1)._stop -490> 2019-03-02 18:53:25.762407 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105942800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=12 - -489> 2019-03-02 18:53:25.762941 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356689 cs=1 l=1).handle_connect_msg accept write reply msg done -488> 2019-03-02 18:53:25.763392 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=356689 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -487> 2019-03-02 18:53:25.763427 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 1 0x560105958000 auth(proto 0 26 bytes epoch 14) v1 -486> 2019-03-02 18:53:25.763441 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (32594381 0 0) 0x560105958000 con 0x560105942800 -485> 2019-03-02 18:53:25.763458 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -484> 2019-03-02 18:53:25.775208 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).read_bulk peer close file descriptor 46 -483> 2019-03-02 18:53:25.775219 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).read_until read failed -482> 2019-03-02 18:53:25.775223 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).process read tag failed -481> 2019-03-02 18:53:25.775226 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1).fault on lossy channel, failing -480> 2019-03-02 18:53:25.775235 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x560105638000 :6789 s=STATE_OPEN pgs=135784 cs=1 l=1)._stop -479> 2019-03-02 18:53:25.775308 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595c000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=46 - -478> 2019-03-02 18:53:25.775369 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135785 cs=1 l=1).handle_connect_msg accept write reply msg done -477> 2019-03-02 18:53:25.775378 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=135785 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -476> 2019-03-02 18:53:25.775413 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 1 0x560105958240 auth(proto 0 27 bytes epoch 14) v1 -475> 2019-03-02 18:53:25.775431 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3015307366 0 0) 0x560105958240 con 0x56010595c000 -474> 2019-03-02 18:53:25.775453 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -473> 2019-03-02 18:53:25.827202 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).read_bulk peer close file descriptor 9 -472> 2019-03-02 18:53:25.827216 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).read_until read failed -471> 2019-03-02 18:53:25.827219 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).process read tag failed -470> 2019-03-02 18:53:25.827223 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1).fault on lossy channel, failing -469> 2019-03-02 18:53:25.827226 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3724611330 conn(0x56010549f000 :6789 s=STATE_OPEN pgs=2902 cs=1 l=1)._stop -468> 2019-03-02 18:53:25.827256 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).read_bulk peer close file descriptor 54 -467> 2019-03-02 18:53:25.827268 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).read_until read failed -466> 2019-03-02 18:53:25.827272 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).process read tag failed -465> 2019-03-02 18:53:25.827283 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1).fault on lossy channel, failing -464> 2019-03-02 18:53:25.827286 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/360631102 conn(0x5601056f4000 :6789 s=STATE_OPEN pgs=2943 cs=1 l=1)._stop -463> 2019-03-02 18:53:25.827414 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595f000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=54 - -462> 2019-03-02 18:53:25.827422 7fe356efc700 1 -- 192.168.253.203:6789/0 >> - conn(0x56010595d800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=9 - -461> 2019-03-02 18:53:25.827490 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105960800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=55 - -460> 2019-03-02 18:53:25.827532 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2952 cs=1 l=1).handle_connect_msg accept write reply msg done -459> 2019-03-02 18:53:25.827574 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3075 cs=1 l=1).handle_connect_msg accept write reply msg done -458> 2019-03-02 18:53:25.827624 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3075 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -457> 2019-03-02 18:53:25.827625 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3017 cs=1 l=1).handle_connect_msg accept write reply msg done -456> 2019-03-02 18:53:25.827648 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=2952 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -455> 2019-03-02 18:53:25.827666 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 1 0x560105435b00 auth(proto 0 34 bytes epoch 14) v1 -454> 2019-03-02 18:53:25.827680 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (3011287488 0 0) 0x560105435b00 con 0x56010595f000 -453> 2019-03-02 18:53:25.827698 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -452> 2019-03-02 18:53:25.827704 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=3017 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -451> 2019-03-02 18:53:25.827710 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 1 0x560105958480 auth(proto 0 34 bytes epoch 14) v1 -450> 2019-03-02 18:53:25.827723 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (2551472749 0 0) 0x560105958480 con 0x56010595d800 -449> 2019-03-02 18:53:25.827732 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -448> 2019-03-02 18:53:25.827747 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 1 0x560105435d40 auth(proto 0 34 bytes epoch 14) v1 -447> 2019-03-02 18:53:25.827767 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 1 ==== auth(proto 0 34 bytes epoch 14) v1 ==== 64+0+0 (133202792 0 0) 0x560105435d40 con 0x560105960800 -446> 2019-03-02 18:53:25.827775 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 34 bytes epoch 14) v1 -445> 2019-03-02 18:53:26.057844 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).read_bulk peer close file descriptor 47 -444> 2019-03-02 18:53:26.057856 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).read_until read failed -443> 2019-03-02 18:53:26.057860 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).process read tag failed -442> 2019-03-02 18:53:26.057863 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1).fault on lossy channel, failing -441> 2019-03-02 18:53:26.057866 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6816/641907 conn(0x560105639800 :6789 s=STATE_OPEN pgs=86317 cs=1 l=1)._stop -440> 2019-03-02 18:53:26.057887 7fe356efc700 1 -- 192.168.253.203:6789/0 reap_dead start -439> 2019-03-02 18:53:26.057897 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x56010549f000 -438> 2019-03-02 18:53:26.057901 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x560105639800 -437> 2019-03-02 18:53:26.057904 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f4000 -436> 2019-03-02 18:53:26.057906 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x5601056f5800 -435> 2019-03-02 18:53:26.057908 7fe356efc700 5 -- 192.168.253.203:6789/0 reap_dead delete 0x560105759800 -434> 2019-03-02 18:53:26.716146 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105972000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=47 - -433> 2019-03-02 18:53:26.716279 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=90617 cs=1 l=1).handle_connect_msg accept write reply msg done -432> 2019-03-02 18:53:26.716312 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=90617 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -431> 2019-03-02 18:53:26.716352 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 1 0x560105492b40 auth(proto 0 26 bytes epoch 14) v1 -430> 2019-03-02 18:53:26.716375 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 1 ==== auth(proto 0 26 bytes epoch 14) v1 ==== 56+0+0 (2372153616 0 0) 0x560105492b40 con 0x560105972000 -429> 2019-03-02 18:53:26.716396 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 26 bytes epoch 14) v1 -428> 2019-03-02 18:53:26.739866 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).read_bulk peer close file descriptor 29 -427> 2019-03-02 18:53:26.739879 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).read_until read failed -426> 2019-03-02 18:53:26.739883 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).process read tag failed -425> 2019-03-02 18:53:26.739887 7fe35d709700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1).fault on lossy channel, failing -424> 2019-03-02 18:53:26.739890 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/2755183883 conn(0x56010563c800 :6789 s=STATE_OPEN pgs=12 cs=1 l=1)._stop -423> 2019-03-02 18:53:26.755304 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).read_bulk peer close file descriptor 57 -422> 2019-03-02 18:53:26.755316 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).read_until read failed -421> 2019-03-02 18:53:26.755320 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).process read tag failed -420> 2019-03-02 18:53:26.755323 7fe356efc700 1 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1).fault on lossy channel, failing -419> 2019-03-02 18:53:26.755326 7fe356efc700 2 -- 192.168.253.203:6789/0 >> 192.168.253.165:6812/710509 conn(0x5601056f8800 :6789 s=STATE_OPEN pgs=411209 cs=1 l=1)._stop -418> 2019-03-02 18:53:26.832047 7fe35d709700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105973800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=29 - -417> 2019-03-02 18:53:26.832645 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=454471 cs=1 l=1).handle_connect_msg accept write reply msg done -416> 2019-03-02 18:53:26.833106 7fe35d709700 2 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=454471 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -415> 2019-03-02 18:53:26.833135 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 1 0x560105492000 auth(proto 0 27 bytes epoch 14) v1 -414> 2019-03-02 18:53:26.833152 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (317843293 0 0) 0x560105492000 con 0x560105973800 -413> 2019-03-02 18:53:26.833170 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -412> 2019-03-02 18:53:26.936709 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> - conn(0x560105975000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=56 - -411> 2019-03-02 18:53:26.936900 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87095 cs=1 l=1).handle_connect_msg accept write reply msg done -410> 2019-03-02 18:53:26.936931 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87095 cs=1 l=1)._process_connection accept get newly_acked_seq 0 -409> 2019-03-02 18:53:26.936962 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87095 cs=1 l=1). rx osd.24 seq 1 0x560105958fc0 auth(proto 0 27 bytes epoch 14) v1 -408> 2019-03-02 18:53:26.936977 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.24 192.168.253.205:6818/642558 1 ==== auth(proto 0 27 bytes epoch 14) v1 ==== 57+0+0 (3667517188 0 0) 0x560105958fc0 con 0x560105975000 -407> 2019-03-02 18:53:26.936994 7fe359f02700 5 mon.storage01roc@3(electing) e14 waitlisting message auth(proto 0 27 bytes epoch 14) v1 -406> 2019-03-02 18:53:26.992061 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).read_bulk peer close file descriptor 28 -405> 2019-03-02 18:53:26.992074 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).read_until read failed -404> 2019-03-02 18:53:26.992078 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).process read tag failed -403> 2019-03-02 18:53:26.992081 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1).fault on lossy channel, failing -402> 2019-03-02 18:53:26.992091 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/1087877254 conn(0x5601054b1800 :6789 s=STATE_OPEN pgs=35503 cs=1 l=1)._stop -401> 2019-03-02 18:53:27.142660 7fe35c707700 5 mon.storage01roc@3(electing).elector(58461) election timer expired -400> 2019-03-02 18:53:27.142900 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105908f00 -399> 2019-03-02 18:53:27.142919 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105908f00 con 0 -398> 2019-03-02 18:53:27.142940 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105909200 -397> 2019-03-02 18:53:27.142946 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105909200 con 0 -396> 2019-03-02 18:53:27.142987 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- ?+0 0x560105909500 -395> 2019-03-02 18:53:27.143002 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- election(484eca8f-c17b-41fd-83eb-314052ccd8ea victory 58462) v7 -- 0x560105909500 con 0 -394> 2019-03-02 18:53:27.143051 7fe35c707700 0 log_channel(cluster) log [INF] : mon.storage01roc is new leader, mons storage01roc,storage02roc,storage03roc,cephmon in quorum (ranks 3,4,5,6) -393> 2019-03-02 18:53:27.143066 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 3 at 2019-03-02 18:53:27.143055) v1 -- 0x560105992000 con 0 -392> 2019-03-02 18:53:27.143107 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 3 at 2019-03-02 18:53:27.143055) v1 ==== 0+0+0 (0 0 0) 0x560105992000 con 0x5601053ea000 -391> 2019-03-02 18:53:27.143292 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x560105909800 -390> 2019-03-02 18:53:27.143304 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x560105909800 con 0 -389> 2019-03-02 18:53:27.143314 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x560105909b00 -388> 2019-03-02 18:53:27.143319 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x560105909b00 con 0 -387> 2019-03-02 18:53:27.143325 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- ?+0 0x56010598e000 -386> 2019-03-02 18:53:27.143329 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(collect lc 13804982 fc 13804454 pn 935303 opn 0) v4 -- 0x56010598e000 con 0 -385> 2019-03-02 18:53:27.143411 7fe35c707700 5 mon.storage01roc@3(leader) e14 apply_quorum_to_compatset_features -384> 2019-03-02 18:53:27.143421 7fe35c707700 5 mon.storage01roc@3(leader) e14 apply_monmap_to_compatset_features -383> 2019-03-02 18:53:27.143486 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105795440 -382> 2019-03-02 18:53:27.143494 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105795440 con 0 -381> 2019-03-02 18:53:27.143502 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105959d40 -380> 2019-03-02 18:53:27.143512 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105959d40 con 0 -379> 2019-03-02 18:53:27.143520 7fe35c707700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- ?+0 0x560105992480 -378> 2019-03-02 18:53:27.143524 7fe35c707700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- time_check( ping e 58462 r 1 ) v1 -- 0x560105992480 con 0 -377> 2019-03-02 18:53:27.143610 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943793 0x5601058a7180 forward(log(1 entries from seq 249 at 2019-03-02 18:53:00.285700) v1 caps allow * tid 3934 con_features 2305244844532236283) v3 -376> 2019-03-02 18:53:27.143638 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943793 ==== forward(log(1 entries from seq 249 at 2019-03-02 18:53:00.285700) v1 caps allow * tid 3934 con_features 2305244844532236283) v3 ==== 451+0+0 (2345767367 0 0) 0x5601058a7180 con 0x560105678800 -375> 2019-03-02 18:53:27.143688 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943794 0x56010599c000 forward(log(1 entries from seq 250 at 2019-03-02 18:53:00.286175) v1 caps allow * tid 3935 con_features 2305244844532236283) v3 -374> 2019-03-02 18:53:27.143715 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943794 ==== forward(log(1 entries from seq 250 at 2019-03-02 18:53:00.286175) v1 caps allow * tid 3935 con_features 2305244844532236283) v3 ==== 521+0+0 (1805236427 0 0) 0x56010599c000 con 0x560105678800 -373> 2019-03-02 18:53:27.144024 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793705 0x560105908f00 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 -372> 2019-03-02 18:53:27.144044 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793705 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (1918868918 0 0) 0x560105908f00 con 0x560105677000 -371> 2019-03-02 18:53:27.144053 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793706 0x560105795440 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144163 ) v1 -370> 2019-03-02 18:53:27.144074 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793706 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144163 ) v1 ==== 36+0+0 (810820296 0 0) 0x560105795440 con 0x560105677000 -369> 2019-03-02 18:53:27.144108 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943795 0x560105909b00 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 -368> 2019-03-02 18:53:27.144128 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943795 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (3193165242 0 0) 0x560105909b00 con 0x560105678800 -367> 2019-03-02 18:53:27.144133 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943796 0x560105959d40 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144374 ) v1 -366> 2019-03-02 18:53:27.144149 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943796 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.144374 ) v1 ==== 36+0+0 (546103651 0 0) 0x560105959d40 con 0x560105678800 -365> 2019-03-02 18:53:27.155849 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216110 0x56010598e000 paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 -364> 2019-03-02 18:53:27.155869 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216110 ==== paxos(last lc 13804982 fc 13804454 pn 935303 opn 0) v4 ==== 84+0+0 (1592508016 0 0) 0x56010598e000 con 0x56010567e000 -363> 2019-03-02 18:53:27.155873 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216111 0x560105992480 time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.154851 ) v1 -362> 2019-03-02 18:53:27.155884 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908c00 -361> 2019-03-02 18:53:27.155889 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908c00 con 0 -360> 2019-03-02 18:53:27.155899 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908600 -359> 2019-03-02 18:53:27.155904 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908600 con 0 -358> 2019-03-02 18:53:27.155911 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- ?+0 0x560105908300 -357> 2019-03-02 18:53:27.155915 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(lease lc 13804982 fc 13804454 pn 0 opn 0) v4 -- 0x560105908300 con 0 -356> 2019-03-02 18:53:27.156054 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943797 0x560105908600 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 -355> 2019-03-02 18:53:27.156055 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793707 0x560105908c00 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 -354> 2019-03-02 18:53:27.156114 7fe359f02700 4 mgrc handle_mgr_map Got map version 519 -353> 2019-03-02 18:53:27.156122 7fe359f02700 4 mgrc handle_mgr_map Active mgr is now 192.168.253.163:6820/2891295 -352> 2019-03-02 18:53:27.156170 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943798 0x56010599c280 forward(log(1 entries from seq 251 at 2019-03-02 18:53:07.139680) v1 caps allow * tid 3940 con_features 2305244844532236283) v3 -351> 2019-03-02 18:53:27.156204 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793708 0x5601058a7400 forward(log(1 entries from seq 352 at 2019-03-02 18:53:07.138814) v1 caps allow * tid 5274 con_features 2305244844532236283) v3 -350> 2019-03-02 18:53:27.156302 7fe359f02700 0 log_channel(cluster) log [DBG] : monmap e14: 7 mons at {cephmon=192.168.253.242:6789/0,storage01moc=192.168.253.163:6789/0,storage01roc=192.168.253.203:6789/0,storage02moc=192.168.253.164:6789/0,storage02roc=192.168.253.204:6789/0,storage03moc=192.168.253.165:6789/0,storage03roc=192.168.253.205:6789/0} -349> 2019-03-02 18:53:27.156309 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 4 at 2019-03-02 18:53:27.156308) v1 -- 0x560105795440 con 0 -348> 2019-03-02 18:53:27.156344 7fe359f02700 0 log_channel(cluster) log [DBG] : fsmap -347> 2019-03-02 18:53:27.156348 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 5 at 2019-03-02 18:53:27.156347) v1 -- 0x560105795b00 con 0 -346> 2019-03-02 18:53:27.156418 7fe359f02700 0 log_channel(cluster) log [DBG] : osdmap e79198: 60 total, 60 up, 60 in -345> 2019-03-02 18:53:27.156422 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 6 at 2019-03-02 18:53:27.156421) v1 -- 0x560105795d40 con 0 -344> 2019-03-02 18:53:27.156482 7fe359f02700 0 log_channel(cluster) log [DBG] : mgrmap e519: storage01moc(active), standbys: storage02roc, storage03roc, storage03moc, storage01roc, cephmon -343> 2019-03-02 18:53:27.156486 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 7 at 2019-03-02 18:53:27.156485) v1 -- 0x560105992900 con 0 -342> 2019-03-02 18:53:27.156679 7fe359f02700 0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 3/7 mons down, quorum storage01roc,storage02roc,storage03roc,cephmon -341> 2019-03-02 18:53:27.156685 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6789/0 -- log(1 entries from seq 8 at 2019-03-02 18:53:27.156683) v1 -- 0x560105992d80 con 0 -340> 2019-03-02 18:53:27.156717 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.242:6789/0 conn(0x56010567e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23438 cs=1 l=0). rx mon.6 seq 1673216112 0x560105908300 paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 -339> 2019-03-02 18:53:27.156915 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601053838c0 0x5601054f1d40 auth_reply(proto 2 0 (0) Success) v1 -338> 2019-03-02 18:53:27.156929 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3361644023 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054f1d40 con 0 -337> 2019-03-02 18:53:27.156965 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601053839e0 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 -336> 2019-03-02 18:53:27.156974 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3589307736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 -335> 2019-03-02 18:53:27.157007 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547a480 0x560105492240 auth_reply(proto 2 0 (0) Success) v1 -334> 2019-03-02 18:53:27.157012 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492240 con 0 -333> 2019-03-02 18:53:27.157029 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547a6c0 0x560105492900 auth_reply(proto 2 0 (0) Success) v1 -332> 2019-03-02 18:53:27.157033 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492900 con 0 -331> 2019-03-02 18:53:27.157060 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547b560 0x560105434000 auth_reply(proto 2 0 (0) Success) v1 -330> 2019-03-02 18:53:27.157065 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/614279813 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434000 con 0 -329> 2019-03-02 18:53:27.157095 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010547bb00 0x5601054938c0 auth_reply(proto 2 0 (0) Success) v1 -328> 2019-03-02 18:53:27.157100 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054938c0 con 0 -327> 2019-03-02 18:53:27.157117 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770000 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 -326> 2019-03-02 18:53:27.157123 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 -325> 2019-03-02 18:53:27.157151 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770120 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 -324> 2019-03-02 18:53:27.157158 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 -323> 2019-03-02 18:53:27.157178 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770240 0x560105434fc0 auth_reply(proto 2 0 (0) Success) v1 -322> 2019-03-02 18:53:27.157181 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434fc0 con 0 -321> 2019-03-02 18:53:27.157196 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770360 0x560105434900 auth_reply(proto 2 0 (0) Success) v1 -320> 2019-03-02 18:53:27.157199 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/94488736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434900 con 0 -319> 2019-03-02 18:53:27.157217 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105770480 0x560105435200 auth_reply(proto 2 0 (0) Success) v1 -318> 2019-03-02 18:53:27.157222 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3202181189 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435200 con 0 -317> 2019-03-02 18:53:27.157217 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 2 0x5601054346c0 auth(proto 2 32 bytes epoch 0) v1 -316> 2019-03-02 18:53:27.157247 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057705a0 0x560105493b00 auth_reply(proto 2 0 (0) Success) v1 -315> 2019-03-02 18:53:27.157250 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3953751394 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493b00 con 0 -314> 2019-03-02 18:53:27.157265 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057706c0 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 -313> 2019-03-02 18:53:27.157267 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/1267131636 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 -312> 2019-03-02 18:53:27.157294 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601057707e0 0x560105794000 auth_reply(proto 2 0 (0) Success) v1 -311> 2019-03-02 18:53:27.157297 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.44:0/219447902 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794000 con 0 -310> 2019-03-02 18:53:27.157317 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105771c20 0x560105794240 auth_reply(proto 2 0 (0) Success) v1 -309> 2019-03-02 18:53:27.157319 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3482224786 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794240 con 0 -308> 2019-03-02 18:53:27.157333 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105771d40 0x560105794fc0 auth_reply(proto 2 0 (0) Success) v1 -307> 2019-03-02 18:53:27.157337 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3137663203 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794fc0 con 0 -306> 2019-03-02 18:53:27.157354 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928000 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 -305> 2019-03-02 18:53:27.157358 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.33:0/140057803 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 -304> 2019-03-02 18:53:27.157360 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 2 0x560105435440 auth(proto 2 32 bytes epoch 0) v1 -303> 2019-03-02 18:53:27.157373 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928120 0x560105794480 auth_reply(proto 2 0 (0) Success) v1 -302> 2019-03-02 18:53:27.157375 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.35:0/3274667345 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794480 con 0 -301> 2019-03-02 18:53:27.157383 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 2 0x560105434fc0 auth(proto 2 144 bytes epoch 0) v1 -300> 2019-03-02 18:53:27.157391 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928360 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 -299> 2019-03-02 18:53:27.157383 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 2 0x560105493b00 auth(proto 2 144 bytes epoch 0) v1 -298> 2019-03-02 18:53:27.157394 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/1733074246 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 -297> 2019-03-02 18:53:27.157406 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928480 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 -296> 2019-03-02 18:53:27.157408 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/2175704180 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 -295> 2019-03-02 18:53:27.157508 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 2 0x5601054358c0 auth(proto 2 144 bytes epoch 0) v1 -294> 2019-03-02 18:53:27.157591 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 2 0x560105795200 auth(proto 2 32 bytes epoch 0) v1 -293> 2019-03-02 18:53:27.157631 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).read_bulk peer close file descriptor 56 -292> 2019-03-02 18:53:27.157647 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).read_until read failed -291> 2019-03-02 18:53:27.157655 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).process read tag failed -290> 2019-03-02 18:53:27.157662 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 2 0x560105435680 auth(proto 2 32 bytes epoch 0) v1 -289> 2019-03-02 18:53:27.157663 7fe3576fd700 1 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1).fault on lossy channel, failing -288> 2019-03-02 18:53:27.157668 7fe3576fd700 2 -- 192.168.253.203:6789/0 >> 192.168.253.205:6818/642558 conn(0x560105975000 :6789 s=STATE_OPEN pgs=87095 cs=1 l=1)._stop -287> 2019-03-02 18:53:27.157709 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 2 0x560105492900 auth(proto 2 144 bytes epoch 0) v1 -286> 2019-03-02 18:53:27.157715 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 2 0x560105434480 auth(proto 2 32 bytes epoch 0) v1 -285> 2019-03-02 18:53:27.157743 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 2 0x5601054938c0 auth(proto 2 144 bytes epoch 0) v1 -284> 2019-03-02 18:53:27.157740 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x560105909b00 -283> 2019-03-02 18:53:27.157746 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 2 0x560105493d40 auth(proto 2 32 bytes epoch 0) v1 -282> 2019-03-02 18:53:27.157754 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x560105909b00 con 0 -281> 2019-03-02 18:53:27.157765 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x56010500af00 -280> 2019-03-02 18:53:27.157769 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x56010500af00 con 0 -279> 2019-03-02 18:53:27.157778 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- ?+0 0x56010598ec00 -278> 2019-03-02 18:53:27.157782 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- paxos(begin lc 13804982 fc 0 pn 935303 opn 0) v4 -- 0x56010598ec00 con 0 -277> 2019-03-02 18:53:27.157810 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059286c0 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 -276> 2019-03-02 18:53:27.157819 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 -275> 2019-03-02 18:53:27.157839 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059287e0 0x560105493200 auth_reply(proto 2 0 (0) Success) v1 -274> 2019-03-02 18:53:27.157844 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493200 con 0 -273> 2019-03-02 18:53:27.157870 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928900 0x560105794900 auth_reply(proto 2 0 (0) Success) v1 -272> 2019-03-02 18:53:27.157876 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794900 con 0 -271> 2019-03-02 18:53:27.157895 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105928a20 0x560105794b40 auth_reply(proto 2 0 (0) Success) v1 -270> 2019-03-02 18:53:27.157900 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794b40 con 0 -269> 2019-03-02 18:53:27.157923 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a900 0x560105958000 auth_reply(proto 2 0 (0) Success) v1 -268> 2019-03-02 18:53:27.157928 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958000 con 0 -267> 2019-03-02 18:53:27.157945 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595aa20 0x560105958240 auth_reply(proto 2 0 (0) Success) v1 -266> 2019-03-02 18:53:27.157949 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/1251637173 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958240 con 0 -265> 2019-03-02 18:53:27.157966 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ab40 0x560105435b00 auth_reply(proto 2 0 (0) Success) v1 -264> 2019-03-02 18:53:27.157970 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/683377982 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435b00 con 0 -263> 2019-03-02 18:53:27.157986 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ac60 0x560105958480 auth_reply(proto 2 0 (0) Success) v1 -262> 2019-03-02 18:53:27.157990 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/419113882 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958480 con 0 -261> 2019-03-02 18:53:27.158007 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595ad80 0x560105435d40 auth_reply(proto 2 0 (0) Success) v1 -260> 2019-03-02 18:53:27.158012 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435d40 con 0 -259> 2019-03-02 18:53:27.158011 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 2 0x5601054926c0 auth(proto 2 32 bytes epoch 0) v1 -258> 2019-03-02 18:53:27.158028 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595aea0 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 -257> 2019-03-02 18:53:27.158033 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 -256> 2019-03-02 18:53:27.158038 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 2 0x560105493200 auth(proto 2 144 bytes epoch 0) v1 -255> 2019-03-02 18:53:27.158061 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 2 0x560105492fc0 auth(proto 2 32 bytes epoch 0) v1 -254> 2019-03-02 18:53:27.158081 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216111 ==== time_check( pong e 58462 r 1 ts 2019-03-02 18:53:27.154851 ) v1 ==== 36+0+0 (282032588 0 0) 0x560105992480 con 0x56010567e000 -253> 2019-03-02 18:53:27.158102 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105958fc0 -252> 2019-03-02 18:53:27.158108 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105958fc0 con 0 -251> 2019-03-02 18:53:27.158116 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105492000 -250> 2019-03-02 18:53:27.158133 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105492000 con 0 -249> 2019-03-02 18:53:27.158140 7fe359f02700 1 -- 192.168.253.203:6789/0 _send_message--> mon.6 192.168.253.242:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- ?+0 0x560105794d80 -248> 2019-03-02 18:53:27.158136 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 2 0x560105958480 auth(proto 2 144 bytes epoch 0) v1 -247> 2019-03-02 18:53:27.158145 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.242:6789/0 -- time_check( report e 58462 r 2 #skews 4 #latencies 4 ) v1 -- 0x560105794d80 con 0 -246> 2019-03-02 18:53:27.158155 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 4 at 2019-03-02 18:53:27.156308) v1 ==== 0+0+0 (0 0 0) 0x560105795440 con 0x5601053ea000 -245> 2019-03-02 18:53:27.158171 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793707 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (1300498542 0 0) 0x560105908c00 con 0x560105677000 -244> 2019-03-02 18:53:27.158168 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6789/0 conn(0x560105677000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9597 cs=1 l=0). rx mon.4 seq 1756793709 0x560105909b00 paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 -243> 2019-03-02 18:53:27.158182 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 2 0x560105958240 auth(proto 2 144 bytes epoch 0) v1 -242> 2019-03-02 18:53:27.158186 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943797 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (613287789 0 0) 0x560105908600 con 0x560105678800 -241> 2019-03-02 18:53:27.158190 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 2 0x560105492b40 auth(proto 2 32 bytes epoch 0) v1 -240> 2019-03-02 18:53:27.158205 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 2 0x560105794fc0 auth(proto 2 144 bytes epoch 0) v1 -239> 2019-03-02 18:53:27.158207 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.6 192.168.253.242:6789/0 1673216112 ==== paxos(lease_ack lc 13804982 fc 13804454 pn 0 opn 0) v4 ==== 182+0+0 (3257341873 0 0) 0x560105908300 con 0x56010567e000 -238> 2019-03-02 18:53:27.158216 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 5 at 2019-03-02 18:53:27.156347) v1 ==== 0+0+0 (0 0 0) 0x560105795b00 con 0x5601053ea000 -237> 2019-03-02 18:53:27.158227 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6789/0 conn(0x560105678800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7287 cs=1 l=0). rx mon.5 seq 1798943799 0x56010500af00 paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 -236> 2019-03-02 18:53:27.158230 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 2 0x560105435d40 auth(proto 2 144 bytes epoch 0) v1 -235> 2019-03-02 18:53:27.158229 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793708 ==== forward(log(1 entries from seq 352 at 2019-03-02 18:53:07.138814) v1 caps allow * tid 5274 con_features 2305244844532236283) v3 ==== 345+0+0 (1198622601 0 0) 0x5601058a7400 con 0x560105677000 -234> 2019-03-02 18:53:27.158250 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 2 0x560105492000 auth(proto 2 144 bytes epoch 0) v1 -233> 2019-03-02 18:53:27.158245 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 2 0x560105958fc0 auth(proto 2 144 bytes epoch 0) v1 -232> 2019-03-02 18:53:27.158260 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943798 ==== forward(log(1 entries from seq 251 at 2019-03-02 18:53:07.139680) v1 caps allow * tid 3940 con_features 2305244844532236283) v3 ==== 345+0+0 (3144552998 0 0) 0x56010599c280 con 0x560105678800 -231> 2019-03-02 18:53:27.158282 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 2 0x560105795680 auth(proto 2 144 bytes epoch 0) v1 -230> 2019-03-02 18:53:27.158287 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 6 at 2019-03-02 18:53:27.156421) v1 ==== 0+0+0 (0 0 0) 0x560105795d40 con 0x5601053ea000 -229> 2019-03-02 18:53:27.158295 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 2 0x560105435200 auth(proto 2 144 bytes epoch 0) v1 -228> 2019-03-02 18:53:27.158300 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.4 192.168.253.204:6789/0 1756793709 ==== paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 ==== 84+0+0 (3782697332 0 0) 0x560105909b00 con 0x560105677000 -227> 2019-03-02 18:53:27.158307 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 2 0x560105492240 auth(proto 2 144 bytes epoch 0) v1 -226> 2019-03-02 18:53:27.158310 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.5 192.168.253.205:6789/0 1798943799 ==== paxos(accept lc 13804982 fc 0 pn 935303 opn 0) v4 ==== 84+0+0 (3694817334 0 0) 0x56010500af00 con 0x560105678800 -225> 2019-03-02 18:53:27.158327 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 7 at 2019-03-02 18:53:27.156485) v1 ==== 0+0+0 (0 0 0) 0x560105992900 con 0x5601053ea000 -224> 2019-03-02 18:53:27.158342 7fe359f02700 1 -- 192.168.253.203:6789/0 <== mon.3 192.168.253.203:6789/0 0 ==== log(1 entries from seq 8 at 2019-03-02 18:53:27.156683) v1 ==== 0+0+0 (0 0 0) 0x560105992d80 con 0x5601053ea000 -223> 2019-03-02 18:53:27.158359 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (3965368829 0 0) 0x5601054346c0 con 0x56010545a000 -222> 2019-03-02 18:53:27.158379 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 2 0x560105493440 auth(proto 2 144 bytes epoch 0) v1 -221> 2019-03-02 18:53:27.158418 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 2 0x560105492480 auth(proto 2 32 bytes epoch 0) v1 -220> 2019-03-02 18:53:27.158453 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 2 0x560105958d80 auth(proto 2 32 bytes epoch 0) v1 -219> 2019-03-02 18:53:27.158475 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 2 0x560105958b40 auth(proto 2 32 bytes epoch 0) v1 -218> 2019-03-02 18:53:27.158491 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105534a20 0x560105992480 auth_reply(proto 2 0 (0) Success) v1 -217> 2019-03-02 18:53:27.158501 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105992480 con 0 -216> 2019-03-02 18:53:27.158517 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1311405791 0 0) 0x5601054358c0 con 0x560105758000 -215> 2019-03-02 18:53:27.158600 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 2 0x560105992480 auth(proto 2 32 bytes epoch 0) v1 -214> 2019-03-02 18:53:27.158615 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105534a20 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 -213> 2019-03-02 18:53:27.158623 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/614279813 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 -212> 2019-03-02 18:53:27.158638 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (538006993 0 0) 0x560105435680 con 0x56010575b000 -211> 2019-03-02 18:53:27.158661 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 3 0x5601054346c0 auth(proto 2 165 bytes epoch 0) v1 -210> 2019-03-02 18:53:27.158701 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 -209> 2019-03-02 18:53:27.158709 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 -208> 2019-03-02 18:53:27.158722 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (147920225 0 0) 0x560105434480 con 0x56010575c800 -207> 2019-03-02 18:53:27.158784 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 -206> 2019-03-02 18:53:27.158792 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 -205> 2019-03-02 18:53:27.158805 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (4035818936 0 0) 0x560105493d40 con 0x560105772000 -204> 2019-03-02 18:53:27.158873 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 -203> 2019-03-02 18:53:27.158881 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 -202> 2019-03-02 18:53:27.158874 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 3 0x5601054da000 mon_subscribe({monmap=15+,osdmap=79199}) v2 -201> 2019-03-02 18:53:27.158893 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (175811679 0 0) 0x560105435440 con 0x560105773800 -200> 2019-03-02 18:53:27.158904 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/614279813 conn(0x560105758000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7222 cs=1 l=1). rx client.8204779 seq 4 0x560105794000 pool_op(delete unmanaged snap pool 116 auid 0 tid 35 name v0) v4 -199> 2019-03-02 18:53:27.158954 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105493d40 auth_reply(proto 2 0 (0) Success) v1 -198> 2019-03-02 18:53:27.158961 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493d40 con 0 -197> 2019-03-02 18:53:27.158975 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2028020967 0 0) 0x560105492900 con 0x560105775000 -196> 2019-03-02 18:53:27.159054 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b7a0 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 -195> 2019-03-02 18:53:27.159062 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/94488736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 -194> 2019-03-02 18:53:27.159075 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (930740453 0 0) 0x560105958240 con 0x560105776800 -193> 2019-03-02 18:53:27.159086 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 3 0x560105493d40 auth(proto 2 165 bytes epoch 0) v1 -192> 2019-03-02 18:53:27.159153 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b8c0 0x560105492900 auth_reply(proto 2 0 (0) Success) v1 -191> 2019-03-02 18:53:27.159161 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3202181189 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492900 con 0 -190> 2019-03-02 18:53:27.159175 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1212249513 0 0) 0x560105958480 con 0x560105788000 -189> 2019-03-02 18:53:27.159252 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059845a0 0x560105958240 auth_reply(proto 2 0 (0) Success) v1 -188> 2019-03-02 18:53:27.159254 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 3 0x560105435680 auth(proto 2 165 bytes epoch 0) v1 -187> 2019-03-02 18:53:27.159262 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/3953751394 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958240 con 0 -186> 2019-03-02 18:53:27.159269 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/94488736 conn(0x560105775000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11207 cs=1 l=1). rx client.8173839 seq 3 0x5601056d8600 mon_subscribe({monmap=15+,osdmap=79199}) v2 -185> 2019-03-02 18:53:27.159283 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3296193308 0 0) 0x5601054938c0 con 0x560105789800 -184> 2019-03-02 18:53:27.159341 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 3 0x5601054358c0 auth(proto 2 181 bytes epoch 0) v1 -183> 2019-03-02 18:53:27.159384 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984480 0x560105958480 auth_reply(proto 2 0 (0) Success) v1 -182> 2019-03-02 18:53:27.159392 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.43:0/1267131636 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958480 con 0 -181> 2019-03-02 18:53:27.159407 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3966250534 0 0) 0x560105435d40 con 0x5601058f8000 -180> 2019-03-02 18:53:27.159466 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 3 0x560105435440 auth(proto 2 165 bytes epoch 0) v1 -179> 2019-03-02 18:53:27.159486 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 2 0x560105434480 auth(proto 2 144 bytes epoch 0) v1 -178> 2019-03-02 18:53:27.159488 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a7e0 0x5601054938c0 auth_reply(proto 2 0 (0) Success) v1 -177> 2019-03-02 18:53:27.159493 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3482224786 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054938c0 con 0 -176> 2019-03-02 18:53:27.159508 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3378494650 0 0) 0x560105492000 con 0x5601058f9800 -175> 2019-03-02 18:53:27.159584 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595b9e0 0x560105435d40 auth_reply(proto 2 0 (0) Success) v1 -174> 2019-03-02 18:53:27.159592 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.31:0/3137663203 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435d40 con 0 -173> 2019-03-02 18:53:27.159606 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1061402963 0 0) 0x560105493200 con 0x5601058fb000 -172> 2019-03-02 18:53:27.159598 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.43:0/1267131636 conn(0x560105789800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6463 cs=1 l=1). rx client.8173846 seq 3 0x560105252400 mon_subscribe({monmap=15+,osdmap=79199}) v2 -171> 2019-03-02 18:53:27.159689 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059847e0 0x560105492000 auth_reply(proto 2 0 (0) Success) v1 -170> 2019-03-02 18:53:27.159697 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.33:0/140057803 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492000 con 0 -169> 2019-03-02 18:53:27.159711 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1382660310 0 0) 0x560105493440 con 0x5601058fc800 -168> 2019-03-02 18:53:27.159786 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535200 0x560105493200 auth_reply(proto 2 0 (0) Success) v1 -167> 2019-03-02 18:53:27.159794 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.35:0/3274667345 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493200 con 0 -166> 2019-03-02 18:53:27.159820 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1173792373 0 0) 0x560105492240 con 0x56010543b800 -165> 2019-03-02 18:53:27.159821 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3202181189 conn(0x560105776800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12571 cs=1 l=1). rx client.9189323 seq 3 0x560105253000 mon_subscribe({monmap=15+,osdmap=79199}) v2 -164> 2019-03-02 18:53:27.159871 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/3953751394 conn(0x560105788000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4540 cs=1 l=1). rx client.9189330 seq 3 0x5601056d9e00 mon_subscribe({monmap=15+,osdmap=79199}) v2 -163> 2019-03-02 18:53:27.159910 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bc20 0x560105493440 auth_reply(proto 2 0 (0) Success) v1 -162> 2019-03-02 18:53:27.159918 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/1733074246 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493440 con 0 -161> 2019-03-02 18:53:27.159932 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (202713985 0 0) 0x560105795680 con 0x56010593e000 -160> 2019-03-02 18:53:27.160010 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535d40 0x560105492240 auth_reply(proto 2 0 (0) Success) v1 -159> 2019-03-02 18:53:27.160017 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.37:0/2175704180 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492240 con 0 -158> 2019-03-02 18:53:27.160031 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (799206973 0 0) 0x560105492480 con 0x56010545b800 -157> 2019-03-02 18:53:27.160092 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 -156> 2019-03-02 18:53:27.160100 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 -155> 2019-03-02 18:53:27.160113 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2241480435 0 0) 0x5601054926c0 con 0x56010593f800 -154> 2019-03-02 18:53:27.160152 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3482224786 conn(0x5601058f8000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18715 cs=1 l=1). rx client.9201939 seq 3 0x560105252c00 mon_subscribe({monmap=15+,osdmap=79199}) v2 -153> 2019-03-02 18:53:27.160172 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.35:0/3274667345 conn(0x5601058fc800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11821 cs=1 l=1). rx client.9191584 seq 3 0x5601056d6000 mon_subscribe({monmap=15+,osdmap=79199}) v2 -152> 2019-03-02 18:53:27.160184 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105492480 auth_reply(proto 2 0 (0) Success) v1 -151> 2019-03-02 18:53:27.160193 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492480 con 0 -150> 2019-03-02 18:53:27.160206 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1127919766 0 0) 0x560105958b40 con 0x560105941000 -149> 2019-03-02 18:53:27.160212 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.31:0/3137663203 conn(0x5601058f9800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13944 cs=1 l=1). rx client.9201946 seq 3 0x5601054da200 mon_subscribe({monmap=15+,osdmap=79199}) v2 -148> 2019-03-02 18:53:27.160270 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 -147> 2019-03-02 18:53:27.160285 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 -146> 2019-03-02 18:53:27.160302 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (3290125480 0 0) 0x560105958d80 con 0x560105942800 -145> 2019-03-02 18:53:27.160350 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 3 0x5601054926c0 auth(proto 2 165 bytes epoch 0) v1 -144> 2019-03-02 18:53:27.160366 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105958b40 auth_reply(proto 2 0 (0) Success) v1 -143> 2019-03-02 18:53:27.160370 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.33:0/140057803 conn(0x5601058fb000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5417 cs=1 l=1). rx client.8943766 seq 3 0x560105253800 mon_subscribe({monmap=15+,osdmap=79199}) v2 -142> 2019-03-02 18:53:27.160374 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958b40 con 0 -141> 2019-03-02 18:53:27.160387 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1138593506 0 0) 0x560105492fc0 con 0x56010595c000 -140> 2019-03-02 18:53:27.160447 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105958d80 auth_reply(proto 2 0 (0) Success) v1 -139> 2019-03-02 18:53:27.160454 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958d80 con 0 -138> 2019-03-02 18:53:27.160468 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2443371689 0 0) 0x560105958fc0 con 0x56010595d800 -137> 2019-03-02 18:53:27.160546 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bb00 0x560105492fc0 auth_reply(proto 2 0 (0) Success) v1 -136> 2019-03-02 18:53:27.160554 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/683377982 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492fc0 con 0 -135> 2019-03-02 18:53:27.160567 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (2901108871 0 0) 0x560105794fc0 con 0x56010595f000 -134> 2019-03-02 18:53:27.160563 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/1733074246 conn(0x56010543b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6891 cs=1 l=1). rx client.8939327 seq 3 0x5601056d7800 mon_subscribe({monmap=15+,osdmap=79199}) v2 -133> 2019-03-02 18:53:27.160590 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 3 0x560105492fc0 auth(proto 2 165 bytes epoch 0) v1 -132> 2019-03-02 18:53:27.160640 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.37:0/2175704180 conn(0x56010593e000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=35760 cs=1 l=1). rx client.8939320 seq 3 0x5601056d9a00 mon_subscribe({monmap=15+,osdmap=79199}) v2 -131> 2019-03-02 18:53:27.160654 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535320 0x560105958fc0 auth_reply(proto 2 0 (0) Success) v1 -130> 2019-03-02 18:53:27.160663 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/1251637173 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958fc0 con 0 -129> 2019-03-02 18:53:27.160664 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 3 0x560105795680 auth(proto 2 165 bytes epoch 0) v1 -128> 2019-03-02 18:53:27.160677 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (558639228 0 0) 0x560105435200 con 0x560105960800 -127> 2019-03-02 18:53:27.160755 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601059846c0 0x560105794fc0 auth_reply(proto 2 0 (0) Success) v1 -126> 2019-03-02 18:53:27.160763 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/419113882 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105794fc0 con 0 -125> 2019-03-02 18:53:27.160776 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (102187718 0 0) 0x560105492b40 con 0x560105972000 -124> 2019-03-02 18:53:27.160779 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/683377982 conn(0x56010595d800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3075 cs=1 l=1). rx client.8775242 seq 3 0x5601056d8800 mon_subscribe({monmap=15+}) v2 -123> 2019-03-02 18:53:27.160841 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105435200 auth_reply(proto 2 0 (0) Success) v1 -122> 2019-03-02 18:53:27.160859 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435200 con 0 -121> 2019-03-02 18:53:27.160864 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 3 0x560105958d80 auth(proto 2 165 bytes epoch 0) v1 -120> 2019-03-02 18:53:27.160876 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2355473505 0 0) 0x560105992480 con 0x560105973800 -119> 2019-03-02 18:53:27.160882 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/1251637173 conn(0x56010595f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2952 cs=1 l=1). rx client.8113023 seq 3 0x5601056d7e00 mon_subscribe({monmap=15+}) v2 -118> 2019-03-02 18:53:27.160925 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 3 0x560105958b40 auth(proto 2 165 bytes epoch 0) v1 -117> 2019-03-02 18:53:27.160942 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 -116> 2019-03-02 18:53:27.160950 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 -115> 2019-03-02 18:53:27.160965 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (199693358 0 0) 0x560105434fc0 con 0x56010548f000 -114> 2019-03-02 18:53:27.161024 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 3 0x560105492b40 auth(proto 2 165 bytes epoch 0) v1 -113> 2019-03-02 18:53:27.161026 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/419113882 conn(0x560105960800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3017 cs=1 l=1). rx client.8665115 seq 3 0x5601056d7400 mon_subscribe({monmap=15+}) v2 -112> 2019-03-02 18:53:27.161043 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535c20 0x560105992480 auth_reply(proto 2 0 (0) Success) v1 -111> 2019-03-02 18:53:27.161049 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3589307736 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105992480 con 0 -110> 2019-03-02 18:53:27.161063 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1048373748 0 0) 0x560105493b00 con 0x560105490800 -109> 2019-03-02 18:53:27.161135 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595a6c0 0x560105434fc0 auth_reply(proto 2 0 (0) Success) v1 -108> 2019-03-02 18:53:27.161143 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.32:0/3361644023 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434fc0 con 0 -107> 2019-03-02 18:53:27.161156 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (1007212614 0 0) 0x560105795200 con 0x56010563b000 -106> 2019-03-02 18:53:27.161216 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984240 0x560105493b00 auth_reply(proto 2 0 (0) Success) v1 -105> 2019-03-02 18:53:27.161224 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493b00 con 0 -104> 2019-03-02 18:53:27.161237 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (2612887804 0 0) 0x5601054346c0 con 0x56010545a000 -103> 2019-03-02 18:53:27.161280 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3589307736 conn(0x56010548f000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3130 cs=1 l=1). rx client.8938298 seq 3 0x5601056d9200 mon_subscribe({monmap=15+}) v2 -102> 2019-03-02 18:53:27.161352 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105984240 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 -101> 2019-03-02 18:53:27.161349 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.32:0/3361644023 conn(0x560105490800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2956 cs=1 l=1). rx client.8665101 seq 3 0x5601056d5000 mon_subscribe({monmap=15+}) v2 -100> 2019-03-02 18:53:27.161361 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 -99> 2019-03-02 18:53:27.161375 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601054da000 con 0x560105758000 -98> 2019-03-02 18:53:27.161398 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.14 192.168.253.164:6808/605500 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (4024092099 0 0) 0x560105435680 con 0x56010575b000 -97> 2019-03-02 18:53:27.161497 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x56010595bd40 0x5601054346c0 auth_reply(proto 2 0 (0) Success) v1 -96> 2019-03-02 18:53:27.161504 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6808/605500 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054346c0 con 0 -95> 2019-03-02 18:53:27.161517 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9267543 192.168.253.163:0/2778817751 3 ==== auth(proto 2 181 bytes epoch 0) v1 ==== 211+0+0 (1957492419 0 0) 0x5601054358c0 con 0x56010575c800 -94> 2019-03-02 18:53:27.161530 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 3 0x560105795200 auth(proto 2 165 bytes epoch 0) v1 -93> 2019-03-02 18:53:27.161561 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 4 0x560104ff7a00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -92> 2019-03-02 18:53:27.161582 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.204:6805/322433 conn(0x56010545a000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95151 cs=1 l=1). rx osd.1 seq 5 0x5601056d5e00 osd_beacon(pgs [109.29,109.9,109.65,109.69] lec 79198 v79198) v1 -91> 2019-03-02 18:53:27.161626 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535560 0x560105435680 auth_reply(proto 2 0 (0) Success) v1 -90> 2019-03-02 18:53:27.161633 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.163:0/2778817751 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435680 con 0 -89> 2019-03-02 18:53:27.161647 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.40 192.168.253.164:6802/541834 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1498713863 0 0) 0x560105435440 con 0x560105772000 -88> 2019-03-02 18:53:27.161737 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535560 0x5601054358c0 auth_reply(proto 2 0 (0) Success) v1 -87> 2019-03-02 18:53:27.161744 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6802/541834 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054358c0 con 0 -86> 2019-03-02 18:53:27.161756 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.3 192.168.253.203:6800/259757 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1354212678 0 0) 0x560105493d40 con 0x560105773800 -85> 2019-03-02 18:53:27.161774 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 3 0x560105493b00 auth(proto 2 165 bytes epoch 0) v1 -84> 2019-03-02 18:53:27.161843 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105545560 0x560105435440 auth_reply(proto 2 0 (0) Success) v1 -83> 2019-03-02 18:53:27.161849 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6800/259757 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105435440 con 0 -82> 2019-03-02 18:53:27.161862 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173839 192.168.253.43:0/94488736 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d8600 con 0x560105775000 -81> 2019-03-02 18:53:27.161876 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189323 192.168.253.37:0/3202181189 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105253000 con 0x560105776800 -80> 2019-03-02 18:53:27.161886 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9189330 192.168.253.37:0/3953751394 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d9e00 con 0x560105788000 -79> 2019-03-02 18:53:27.161896 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8173846 192.168.253.43:0/1267131636 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105252400 con 0x560105789800 -78> 2019-03-02 18:53:27.161906 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.7893124 192.168.253.44:0/219447902 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (1800022541 0 0) 0x560105434480 con 0x56010578b000 -77> 2019-03-02 18:53:27.161982 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535440 0x560105493d40 auth_reply(proto 2 0 (0) Success) v1 -76> 2019-03-02 18:53:27.161990 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.44:0/219447902 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105493d40 con 0 -75> 2019-03-02 18:53:27.162004 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201939 192.168.253.31:0/3482224786 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105252c00 con 0x5601058f8000 -74> 2019-03-02 18:53:27.162009 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 4 0x5601056d6c00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -73> 2019-03-02 18:53:27.162018 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9201946 192.168.253.31:0/3137663203 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601054da200 con 0x5601058f9800 -72> 2019-03-02 18:53:27.162027 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8943766 192.168.253.33:0/140057803 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x560105253800 con 0x5601058fb000 -71> 2019-03-02 18:53:27.162030 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 5 0x5601056d6a00 osd_beacon(pgs [109.44,109.5c,109.37,109.4a,109.74,109.5f] lec 79198 v79198) v1 -70> 2019-03-02 18:53:27.162036 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.9191584 192.168.253.35:0/3274667345 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d6000 con 0x5601058fc800 -69> 2019-03-02 18:53:27.162042 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6800/259757 conn(0x560105773800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=89395 cs=1 l=1). rx osd.3 seq 6 0x5601056d9e00 osd_beacon(pgs [] lec 79198 v79198) v1 -68> 2019-03-02 18:53:27.162045 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939327 192.168.253.37:0/1733074246 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d7800 con 0x56010543b800 -67> 2019-03-02 18:53:27.162052 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8939320 192.168.253.37:0/2175704180 3 ==== mon_subscribe({monmap=15+,osdmap=79199}) v2 ==== 42+0+0 (179502811 0 0) 0x5601056d9a00 con 0x56010593e000 -66> 2019-03-02 18:53:27.162060 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.19 192.168.253.164:6810/343964 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1446845268 0 0) 0x560105795680 con 0x56010545b800 -65> 2019-03-02 18:53:27.162082 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 4 0x5601056d8600 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -64> 2019-03-02 18:53:27.162097 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6808/605500 conn(0x56010575b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189842 cs=1 l=1). rx osd.14 seq 5 0x560105253000 osd_beacon(pgs [105.c,105.1c8,116.78,105.109,105.d7,105.64,105.297,85.ff,116.17,115.10,105.108,85.6f,85.a7,105.2e0,105.329,105.2ae,105.253,119.37,85.aa,105.19f,105.d6,105.73,115.13,85.1a,105.310,85.98,105.6f,105.97,85.6,121.6f,116.e6,105.1f8,105.31f,105.2ea,105.1d5,105.100,121.30,105.105,121.56,85.7a,105.e4,121.79,121.18,105.18c,108.1d,116.1a,121.33,119.3d,105.3c,116.f0,105.261,105.21a,85.b0,105.1f4,116.7b,85.84,105.4,85.38,115.c,119.8,105.30,105.11d,121.3e,108.2b,105.1bc,105.190,105.36,116.99,105.290,85.18,85.e6,116.c7,116.4b,105.2fd,85.a5,115.e,105.16b,116.f6,105.3f7,85.8a] lec 79197 v79198) v1 -63> 2019-03-02 18:53:27.162152 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105545680 0x560105434480 auth_reply(proto 2 0 (0) Success) v1 -62> 2019-03-02 18:53:27.162160 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6810/343964 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105434480 con 0 -61> 2019-03-02 18:53:27.162173 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.2 192.168.253.205:6804/639792 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3437510617 0 0) 0x5601054926c0 con 0x56010593f800 -60> 2019-03-02 18:53:27.162259 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055457a0 0x560105795680 auth_reply(proto 2 0 (0) Success) v1 -59> 2019-03-02 18:53:27.162265 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6804/639792 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795680 con 0 -58> 2019-03-02 18:53:27.162293 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 4 0x560105252400 mon_subscribe({fsmap=2+,log-info=0+,mgrdigest=0+,mgrmap=0+,monmap=15+,osdmap=79199,servicemap=0+}) v2 -57> 2019-03-02 18:53:27.162314 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.50 192.168.253.165:6816/711134 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1117087331 0 0) 0x560105958d80 con 0x560105941000 -56> 2019-03-02 18:53:27.162316 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 5 0x56010598fb00 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 -55> 2019-03-02 18:53:27.162335 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 6 0x56010599cf00 monmgrreport(0 checks) v1 -54> 2019-03-02 18:53:27.162352 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 7 0x56010598f800 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 -53> 2019-03-02 18:53:27.162361 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 4 0x5601056d9800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -52> 2019-03-02 18:53:27.162367 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 8 0x5601058a7180 monmgrreport(0 checks) v1 -51> 2019-03-02 18:53:27.162388 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 9 0x56010598f500 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 -50> 2019-03-02 18:53:27.162385 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 5 0x5601056d4a00 osd_beacon(pgs [121.5f,121.22,121.f,105.2b7,121.4f,105.2e2,105.110,105.db,119.10,119.2b,105.ee,105.2cb,105.2b,105.86,105.3b,105.2ce,105.3b4,85.50,105.d1,115.8,115.32,85.d9,105.ca,105.ef,108.0,105.fa,119.21,85.3,119.20,105.c0,116.3c,105.80,108.4e,85.bc,105.1dc,105.148,116.df,116.a1,116.82,105.3fd,116.63,116.a4,105.6d,105.0,105.19e,116.4d,105.18f,116.d3,121.3c,105.33a,116.a8,108.79,105.8c,85.4b,105.1cb,105.33f,105.1b6,105.3ab,105.188,105.2dc,108.3c,105.132] lec 79197 v79198) v1 -49> 2019-03-02 18:53:27.162403 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6802/541834 conn(0x560105772000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=279118 cs=1 l=1). rx osd.40 seq 6 0x5601056d4200 osd_beacon(pgs [] lec 79197 v79198) v1 -48> 2019-03-02 18:53:27.162407 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 10 0x56010599d400 monmgrreport(0 checks) v1 -47> 2019-03-02 18:53:27.162411 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x560105535680 0x5601054926c0 auth_reply(proto 2 0 (0) Success) v1 -46> 2019-03-02 18:53:27.162416 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6816/711134 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x5601054926c0 con 0 -45> 2019-03-02 18:53:27.162428 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.9 192.168.253.165:6802/708867 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (2825895776 0 0) 0x560105958b40 con 0x560105942800 -44> 2019-03-02 18:53:27.162516 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055358c0 0x560105958d80 auth_reply(proto 2 0 (0) Success) v1 -43> 2019-03-02 18:53:27.162523 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6802/708867 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958d80 con 0 -42> 2019-03-02 18:53:27.162535 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.35 192.168.253.203:6802/506915 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3530128003 0 0) 0x560105492fc0 con 0x56010595c000 -41> 2019-03-02 18:53:27.162563 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 4 0x5601056d7200 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -40> 2019-03-02 18:53:27.162575 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6804/639792 conn(0x56010593f800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=68711 cs=1 l=1). rx osd.2 seq 5 0x5601056d5600 osd_beacon(pgs [109.2b,109.1f,109.76,109.28,109.1e,109.5] lec 79198 v79198) v1 -39> 2019-03-02 18:53:27.162624 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055359e0 0x560105958b40 auth_reply(proto 2 0 (0) Success) v1 -38> 2019-03-02 18:53:27.162630 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.203:6802/506915 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105958b40 con 0 -37> 2019-03-02 18:53:27.162643 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8775242 192.168.253.32:0/683377982 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d8800 con 0x56010595d800 -36> 2019-03-02 18:53:27.162655 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8113023 192.168.253.32:0/1251637173 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d7e00 con 0x56010595f000 -35> 2019-03-02 18:53:27.162665 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665115 192.168.253.32:0/419113882 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d7400 con 0x560105960800 -34> 2019-03-02 18:53:27.162674 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.7 192.168.253.205:6812/640516 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1436488630 0 0) 0x560105492b40 con 0x560105972000 -33> 2019-03-02 18:53:27.162776 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055357a0 0x560105492fc0 auth_reply(proto 2 0 (0) Success) v1 -32> 2019-03-02 18:53:27.162775 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 11 0x560105909800 mgrbeacon mgr.storage01moc(484eca8f-c17b-41fd-83eb-314052ccd8ea,9267543, 192.168.253.163:6820/2891295, 1) v6 -31> 2019-03-02 18:53:27.162780 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 4 0x5601056d7800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -30> 2019-03-02 18:53:27.162786 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.205:6812/640516 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492fc0 con 0 -29> 2019-03-02 18:53:27.162797 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.163:0/2778817751 conn(0x56010575c800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=3778 cs=1 l=1). rx client.9267543 seq 12 0x56010599db80 monmgrreport(0 checks) v1 -28> 2019-03-02 18:53:27.162799 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.52 192.168.253.164:6812/543269 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (1132521040 0 0) 0x560105795200 con 0x560105973800 -27> 2019-03-02 18:53:27.162798 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6810/343964 conn(0x56010545b800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=330841 cs=1 l=1). rx osd.19 seq 5 0x5601056d7e00 osd_beacon(pgs [121.47,121.29,121.e,105.223,105.298,105.4b,116.91,105.7e,105.170,105.3b7,105.137,105.27b,105.368,85.17,105.226,108.1,105.11a,105.122,105.2f3,105.85,115.2a,105.18e,105.27,85.10,85.54,105.192,85.37,116.6,108.7a,85.97,116.a2,105.370,116.65,116.cf,119.23,116.d4,108.3e,121.7d,85.df,105.e3,105.138,85.e2,105.20f,108.56,85.ab,85.62,105.1b9,105.e,105.26c,105.294,115.3a,105.246,105.2e4,105.1bd,105.28e,105.d9,105.134,105.3b5,105.23a,105.14b,105.293,85.12,105.3d9,105.210] lec 79197 v79198) v1 -26> 2019-03-02 18:53:27.162810 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 4 0x5601056d4000 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -25> 2019-03-02 18:53:27.162818 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.203:6802/506915 conn(0x56010595c000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=135785 cs=1 l=1). rx osd.35 seq 5 0x560105252c00 osd_beacon(pgs [109.52,109.3d] lec 79198 v79198) v1 -24> 2019-03-02 18:53:27.162842 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.44:0/219447902 conn(0x56010578b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54339 cs=1 l=1). rx client.7893124 seq 3 0x5601056d4600 mon_subscribe({monmap=15+,osdmap=79199}) v2 -23> 2019-03-02 18:53:27.162891 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055458c0 0x560105492b40 auth_reply(proto 2 0 (0) Success) v1 -22> 2019-03-02 18:53:27.162898 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.164:6812/543269 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105492b40 con 0 -21> 2019-03-02 18:53:27.162912 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8938298 192.168.253.32:0/3589307736 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d9200 con 0x56010548f000 -20> 2019-03-02 18:53:27.162925 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8665101 192.168.253.32:0/3361644023 3 ==== mon_subscribe({monmap=15+}) v2 ==== 23+0+0 (529233479 0 0) 0x5601056d5000 con 0x560105490800 -19> 2019-03-02 18:53:27.162935 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.15 192.168.253.165:6808/526512 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (399444511 0 0) 0x560105493b00 con 0x56010563b000 -18> 2019-03-02 18:53:27.162987 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 4 0x5601056d7400 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -17> 2019-03-02 18:53:27.163009 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 5 0x5601056d8800 osd_beacon(pgs [109.71,109.66,109.31,109.41] lec 79198 v79198) v1 -16> 2019-03-02 18:53:27.163017 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 4 0x560105253800 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -15> 2019-03-02 18:53:27.163023 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.205:6812/640516 conn(0x560105972000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=90617 cs=1 l=1). rx osd.7 seq 6 0x5601056d5800 osd_beacon(pgs [] lec 79198 v79198) v1 -14> 2019-03-02 18:53:27.163029 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 5 0x5601054da200 osd_beacon(pgs [121.50,121.8,105.e2,121.74,105.1f2,105.346,105.2,105.1ee,105.24a,105.1af,105.187,105.302,105.3bd,121.5d,108.7c,105.dc,116.d9,105.1e2,116.ba,85.d1,116.5c,85.7d,105.cf,105.180,116.2f,116.46,115.0,116.66,85.47,105.2f8,116.d8,116.ed,105.379,108.5,105.ea,105.1d9,116.9d,116.f4,105.3a,85.e1,116.bb,108.4a,116.fb,121.31,115.3b,116.45,105.2ef,105.34f,85.1c,85.96,105.1ea,116.c8,105.263,116.53,105.32a,108.22,105.c6,108.77,105.397,105.348,108.1a] lec 79197 v79198) v1 -13> 2019-03-02 18:53:27.163034 7fe359f02700 2 mon.storage01roc@3(leader) e14 send_reply 0x5601055459e0 0x560105795200 auth_reply(proto 2 0 (0) Success) v1 -12> 2019-03-02 18:53:27.163046 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6816/711134 conn(0x560105941000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=245753 cs=1 l=1). rx osd.50 seq 6 0x5601054da000 osd_beacon(pgs [] lec 79197 v79198) v1 -11> 2019-03-02 18:53:27.163044 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.165:6808/526512 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x560105795200 con 0 -10> 2019-03-02 18:53:27.163061 7fe359f02700 1 -- 192.168.253.203:6789/0 <== osd.1 192.168.253.204:6805/322433 4 ==== mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 ==== 69+0+0 (2557949307 0 0) 0x560104ff7a00 con 0x56010545a000 -9> 2019-03-02 18:53:27.163095 7fe359f02700 1 -- 192.168.253.203:6789/0 --> 192.168.253.204:6805/322433 -- mgrmap(e 519) v1 -- 0x5601056b02c0 con 0 -8> 2019-03-02 18:53:27.163113 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 4 0x5601054da600 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -7> 2019-03-02 18:53:27.163112 7fe359f02700 1 -- 192.168.253.203:6789/0 <== client.8204779 192.168.253.32:0/614279813 4 ==== pool_op(delete unmanaged snap pool 116 auid 0 tid 35 name v0) v4 ==== 65+0+0 (3422141634 0 0) 0x560105794000 con 0x560105758000 -6> 2019-03-02 18:53:27.163129 7fe356efc700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6802/708867 conn(0x560105942800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=356689 cs=1 l=1). rx osd.9 seq 5 0x5601054dba00 osd_beacon(pgs [121.77,105.12d,116.5f,85.a4,85.14,85.b1,116.a,85.4d,116.8a,105.177,115.24,105.3f3,105.336,85.60,105.232,108.7d,85.75,108.5f,85.ee,105.65,105.1fc,105.3b9,116.de,105.c3,108.9,85.f4,105.22c,105.60,116.ad,105.39c,108.58,121.6a,116.67,121.3b,116.9f,105.1db,85.74,105.23,105.36b,105.211,116.e3,105.289,105.12e,105.7f,115.1a,121.51,105.35d,116.76,105.392,105.a0,105.3fc,108.20,85.ac,85.eb,105.10b,105.3d5,121.c,105.26a,105.3e3,85.b6,105.1ca,105.3d6,108.39,105.388,116.ec,115.3d,105.3b0,105.145,105.2ac,115.5,116.30,108.28] lec 79197 v79198) v1 -5> 2019-03-02 18:53:27.163517 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 4 0x5601056d9a00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -4> 2019-03-02 18:53:27.163537 7fe35d709700 5 -- 192.168.253.203:6789/0 >> 192.168.253.164:6812/543269 conn(0x560105973800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=454471 cs=1 l=1). rx osd.52 seq 5 0x5601056d4c00 osd_beacon(pgs [121.7b,121.5c,85.ca,105.295,105.2b0,105.1a3,105.327,105.304,105.3e6,105.1e0,105.19b,85.90,105.81,85.1f,105.2bc,119.18,105.1c7,105.2f4,105.21d,105.3af,105.3f2,116.26,116.27,116.c0,85.e7,105.39e,105.44,116.5,105.143,116.1c,116.ae,105.2e6,105.26d,116.b0,105.1b3,105.3cb,105.d2,108.1c,105.2a3,116.75,105.d4,105.279,105.37c,105.a9,105.a3,85.56,105.3a4,85.63,85.c2,105.74,105.31b,116.6d,105.34e,105.212,108.65,105.1fe] lec 79197 v79198) v1 -3> 2019-03-02 18:53:27.163637 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 4 0x5601056d9c00 mon_subscribe({mgrmap=0+,monmap=15+,osd_pg_creates=0+}) v2 -2> 2019-03-02 18:53:27.163654 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 5 0x5601056d6e00 osd_beacon(pgs [121.7a,121.3a,121.d,115.2e,105.fe,105.2b6,105.2b2,105.9b,85.8b,119.0,105.272,105.3dd,116.60,105.51,105.372,105.29e,105.350,85.53,105.331,105.28b,85.4,105.9f,116.ff,85.15,121.5,116.8,115.f,108.10,116.7a,105.158,105.213,116.b7,105.238,116.16,105.333,85.fb,116.e8,105.1ed,85.9f,105.31d,105.1b2,105.224,85.8f,108.15,116.d,105.be,85.24,115.21,105.173,105.369,85.f1,105.364,105.236,105.f7,85.92,108.35,119.1f,105.296,105.11e,105.2ca,105.167,105.de,105.334,116.2a,105.3c4,105.3e7,105.40] lec 79197 v79198) v1 -1> 2019-03-02 18:53:27.163666 7fe3576fd700 5 -- 192.168.253.203:6789/0 >> 192.168.253.165:6808/526512 conn(0x56010563b000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=465203 cs=1 l=1). rx osd.15 seq 6 0x5601056d6800 osd_beacon(pgs [] lec 79197 v79198) v1 0> 2019-03-02 18:53:27.166322 7fe359f02700 -1 *** Caught signal (Aborted) ** in thread 7fe359f02700 thread_name:ms_dispatch ceph version 12.2.5-2-ge988fb6 (e988fb6c5457b9dd6f1e11ce1b5fd58b4c3a828c) luminous (stable) 1: (()+0x8f53b1) [0x5600fb5d83b1] 2: (()+0xf6d0) [0x7fe3634736d0] 3: (gsignal()+0x37) [0x7fe3607ae277] 4: (abort()+0x148) [0x7fe3607af968] 5: (std::_Rb_tree_iterator, std::allocator > > > > std::_Rb_tree, std::allocator > > >, std::_Select1st, std::allocator > > > >, std::less, std::allocator, std::allocator > > > > >::_M_emplace_hint_unique, std::allocator > > > >(std::_Rb_tree_const_iterator, std::allocator > > > >, std::pair, std::allocator > > >&&)+0) [0x5600fb420780] 6: (pg_pool_t::remove_unmanaged_snap(snapid_t)+0x4d) [0x5600fb40d2dd] 7: (OSDMonitor::prepare_pool_op(boost::intrusive_ptr)+0xe40) [0x5600fb200f10] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x336) [0x5600fb2388c6] 9: (PaxosService::dispatch(boost::intrusive_ptr)+0xaf8) [0x5600fb1c4858] 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x5af) [0x5600fb0a4fef] 11: (Monitor::_ms_dispatch(Message*)+0x7eb) [0x5600fb0a65db] 12: (Monitor::ms_dispatch(Message*)+0x23) [0x5600fb0d2743] 13: (DispatchQueue::entry()+0x792) [0x5600fb583752] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x5600fb37c90d] 15: (()+0x7e25) [0x7fe36346be25] 16: (clone()+0x6d) [0x7fe360876bad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 5/ 5 ms 5/ 5 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.storage01roc.log --- end dump of recent events ---