Project

General

Profile

ceph-mon.smp-016.log

mon log - Radosław Piliszek, 02/13/2018 08:01 AM

Download (54 KB)

 
1
2018-02-13 08:52:03.069170 7f184e9f9e40  0 set uid:gid to 167:167 (ceph:ceph)
2
2018-02-13 08:52:03.069200 7f184e9f9e40  0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 16325
3
2018-02-13 08:52:03.069427 7f184e9f9e40  0 pidfile_write: ignore empty --pid-file
4
2018-02-13 08:52:03.112586 7f184e9f9e40  0 load: jerasure load: lrc load: isa 
5
2018-02-13 08:52:03.112774 7f184e9f9e40  0  set rocksdb option compression = kNoCompression
6
2018-02-13 08:52:03.112788 7f184e9f9e40  0  set rocksdb option write_buffer_size = 33554432
7
2018-02-13 08:52:03.112824 7f184e9f9e40  0  set rocksdb option compression = kNoCompression
8
2018-02-13 08:52:03.112832 7f184e9f9e40  0  set rocksdb option write_buffer_size = 33554432
9
2018-02-13 08:52:03.113012 7f184e9f9e40  4 rocksdb: RocksDB version: 5.4.0
10

    
11
2018-02-13 08:52:03.113024 7f184e9f9e40  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
12
2018-02-13 08:52:03.113026 7f184e9f9e40  4 rocksdb: Compile date Jan 12 2018
13
2018-02-13 08:52:03.113031 7f184e9f9e40  4 rocksdb: DB SUMMARY
14

    
15
2018-02-13 08:52:03.113111 7f184e9f9e40  4 rocksdb: CURRENT file:  CURRENT
16

    
17
2018-02-13 08:52:03.113117 7f184e9f9e40  4 rocksdb: IDENTITY file:  IDENTITY
18

    
19
2018-02-13 08:52:03.113122 7f184e9f9e40  4 rocksdb: MANIFEST file:  MANIFEST-000101 size: 214 Bytes
20

    
21
2018-02-13 08:52:03.113126 7f184e9f9e40  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smp-016/store.db dir, Total Num: 3, files: 000081.sst 000097.sst 000100.sst 
22

    
23
2018-02-13 08:52:03.113128 7f184e9f9e40  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smp-016/store.db: 000102.log size: 18056 ; 
24

    
25
2018-02-13 08:52:03.113131 7f184e9f9e40  4 rocksdb:                         Options.error_if_exists: 0
26
2018-02-13 08:52:03.113132 7f184e9f9e40  4 rocksdb:                       Options.create_if_missing: 0
27
2018-02-13 08:52:03.113133 7f184e9f9e40  4 rocksdb:                         Options.paranoid_checks: 1
28
2018-02-13 08:52:03.113134 7f184e9f9e40  4 rocksdb:                                     Options.env: 0x5565f834f040
29
2018-02-13 08:52:03.113136 7f184e9f9e40  4 rocksdb:                                Options.info_log: 0x5565f98abc80
30
2018-02-13 08:52:03.113137 7f184e9f9e40  4 rocksdb:                          Options.max_open_files: -1
31
2018-02-13 08:52:03.113138 7f184e9f9e40  4 rocksdb:                Options.max_file_opening_threads: 16
32
2018-02-13 08:52:03.113139 7f184e9f9e40  4 rocksdb:                               Options.use_fsync: 0
33
2018-02-13 08:52:03.113139 7f184e9f9e40  4 rocksdb:                       Options.max_log_file_size: 0
34
2018-02-13 08:52:03.113140 7f184e9f9e40  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
35
2018-02-13 08:52:03.113141 7f184e9f9e40  4 rocksdb:                   Options.log_file_time_to_roll: 0
36
2018-02-13 08:52:03.113142 7f184e9f9e40  4 rocksdb:                       Options.keep_log_file_num: 1000
37
2018-02-13 08:52:03.113143 7f184e9f9e40  4 rocksdb:                    Options.recycle_log_file_num: 0
38
2018-02-13 08:52:03.113144 7f184e9f9e40  4 rocksdb:                         Options.allow_fallocate: 1
39
2018-02-13 08:52:03.113145 7f184e9f9e40  4 rocksdb:                        Options.allow_mmap_reads: 0
40
2018-02-13 08:52:03.113146 7f184e9f9e40  4 rocksdb:                       Options.allow_mmap_writes: 0
41
2018-02-13 08:52:03.113147 7f184e9f9e40  4 rocksdb:                        Options.use_direct_reads: 0
42
2018-02-13 08:52:03.113149 7f184e9f9e40  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
43
2018-02-13 08:52:03.113150 7f184e9f9e40  4 rocksdb:          Options.create_missing_column_families: 0
44
2018-02-13 08:52:03.113151 7f184e9f9e40  4 rocksdb:                              Options.db_log_dir: 
45
2018-02-13 08:52:03.113152 7f184e9f9e40  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-smp-016/store.db
46
2018-02-13 08:52:03.113153 7f184e9f9e40  4 rocksdb:                Options.table_cache_numshardbits: 6
47
2018-02-13 08:52:03.113154 7f184e9f9e40  4 rocksdb:                      Options.max_subcompactions: 1
48
2018-02-13 08:52:03.113155 7f184e9f9e40  4 rocksdb:                  Options.max_background_flushes: 1
49
2018-02-13 08:52:03.113155 7f184e9f9e40  4 rocksdb:                         Options.WAL_ttl_seconds: 0
50
2018-02-13 08:52:03.113156 7f184e9f9e40  4 rocksdb:                       Options.WAL_size_limit_MB: 0
51
2018-02-13 08:52:03.113157 7f184e9f9e40  4 rocksdb:             Options.manifest_preallocation_size: 4194304
52
2018-02-13 08:52:03.113158 7f184e9f9e40  4 rocksdb:                     Options.is_fd_close_on_exec: 1
53
2018-02-13 08:52:03.113159 7f184e9f9e40  4 rocksdb:                   Options.advise_random_on_open: 1
54
2018-02-13 08:52:03.113160 7f184e9f9e40  4 rocksdb:                    Options.db_write_buffer_size: 0
55
2018-02-13 08:52:03.113161 7f184e9f9e40  4 rocksdb:         Options.access_hint_on_compaction_start: 1
56
2018-02-13 08:52:03.113162 7f184e9f9e40  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 0
57
2018-02-13 08:52:03.113162 7f184e9f9e40  4 rocksdb:               Options.compaction_readahead_size: 0
58
2018-02-13 08:52:03.113163 7f184e9f9e40  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
59
2018-02-13 08:52:03.113164 7f184e9f9e40  4 rocksdb:           Options.writable_file_max_buffer_size: 1048576
60
2018-02-13 08:52:03.113165 7f184e9f9e40  4 rocksdb:                      Options.use_adaptive_mutex: 0
61
2018-02-13 08:52:03.113165 7f184e9f9e40  4 rocksdb:                            Options.rate_limiter: (nil)
62
2018-02-13 08:52:03.113167 7f184e9f9e40  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
63
2018-02-13 08:52:03.113167 7f184e9f9e40  4 rocksdb:                          Options.bytes_per_sync: 0
64
2018-02-13 08:52:03.113168 7f184e9f9e40  4 rocksdb:                      Options.wal_bytes_per_sync: 0
65
2018-02-13 08:52:03.113169 7f184e9f9e40  4 rocksdb:                       Options.wal_recovery_mode: 2
66
2018-02-13 08:52:03.113170 7f184e9f9e40  4 rocksdb:                  Options.enable_thread_tracking: 0
67
2018-02-13 08:52:03.113171 7f184e9f9e40  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
68
2018-02-13 08:52:03.113172 7f184e9f9e40  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
69
2018-02-13 08:52:03.113172 7f184e9f9e40  4 rocksdb:             Options.write_thread_max_yield_usec: 100
70
2018-02-13 08:52:03.113173 7f184e9f9e40  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
71
2018-02-13 08:52:03.113174 7f184e9f9e40  4 rocksdb:                               Options.row_cache: None
72
2018-02-13 08:52:03.113175 7f184e9f9e40  4 rocksdb:                              Options.wal_filter: None
73
2018-02-13 08:52:03.113176 7f184e9f9e40  4 rocksdb:             Options.avoid_flush_during_recovery: 0
74
2018-02-13 08:52:03.113177 7f184e9f9e40  4 rocksdb:             Options.base_background_compactions: 1
75
2018-02-13 08:52:03.113177 7f184e9f9e40  4 rocksdb:             Options.max_background_compactions: 1
76
2018-02-13 08:52:03.113178 7f184e9f9e40  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
77
2018-02-13 08:52:03.113179 7f184e9f9e40  4 rocksdb:             Options.delayed_write_rate : 16777216
78
2018-02-13 08:52:03.113180 7f184e9f9e40  4 rocksdb:             Options.max_total_wal_size: 0
79
2018-02-13 08:52:03.113181 7f184e9f9e40  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
80
2018-02-13 08:52:03.113182 7f184e9f9e40  4 rocksdb:                   Options.stats_dump_period_sec: 600
81
2018-02-13 08:52:03.113183 7f184e9f9e40  4 rocksdb: Compression algorithms supported:
82
2018-02-13 08:52:03.113187 7f184e9f9e40  4 rocksdb: 	Snappy supported: 0
83
2018-02-13 08:52:03.113199 7f184e9f9e40  4 rocksdb: 	Zlib supported: 0
84
2018-02-13 08:52:03.113200 7f184e9f9e40  4 rocksdb: 	Bzip supported: 0
85
2018-02-13 08:52:03.113201 7f184e9f9e40  4 rocksdb: 	LZ4 supported: 0
86
2018-02-13 08:52:03.113202 7f184e9f9e40  4 rocksdb: 	ZSTD supported: 0
87
2018-02-13 08:52:03.113204 7f184e9f9e40  4 rocksdb: Fast CRC32 supported: 0
88
2018-02-13 08:52:03.113510 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000101
89

    
90
2018-02-13 08:52:03.113648 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
91

    
92
2018-02-13 08:52:03.113657 7f184e9f9e40  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
93
2018-02-13 08:52:03.113659 7f184e9f9e40  4 rocksdb:           Options.merge_operator: 
94
2018-02-13 08:52:03.113660 7f184e9f9e40  4 rocksdb:        Options.compaction_filter: None
95
2018-02-13 08:52:03.113661 7f184e9f9e40  4 rocksdb:        Options.compaction_filter_factory: None
96
2018-02-13 08:52:03.113662 7f184e9f9e40  4 rocksdb:         Options.memtable_factory: SkipListFactory
97
2018-02-13 08:52:03.113662 7f184e9f9e40  4 rocksdb:            Options.table_factory: BlockBasedTable
98
2018-02-13 08:52:03.113684 7f184e9f9e40  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565f95de158)
99
  cache_index_and_filter_blocks: 1
100
  cache_index_and_filter_blocks_with_high_priority: 1
101
  pin_l0_filter_and_index_blocks_in_cache: 1
102
  index_type: 0
103
  hash_index_allow_collision: 1
104
  checksum: 1
105
  no_block_cache: 0
106
  block_cache: 0x5565f96009a8
107
  block_cache_name: LRUCache
108
  block_cache_options:
109
    capacity : 134217728
110
    num_shard_bits : 4
111
    strict_capacity_limit : 0
112
    high_pri_pool_ratio: 0.000
113
  block_cache_compressed: (nil)
114
  persistent_cache: (nil)
115
  block_size: 4096
116
  block_size_deviation: 10
117
  block_restart_interval: 16
118
  index_block_restart_interval: 1
119
  filter_policy: rocksdb.BuiltinBloomFilter
120
  whole_key_filtering: 1
121
  format_version: 2
122

    
123
2018-02-13 08:52:03.113712 7f184e9f9e40  4 rocksdb:        Options.write_buffer_size: 33554432
124
2018-02-13 08:52:03.113716 7f184e9f9e40  4 rocksdb:  Options.max_write_buffer_number: 2
125
2018-02-13 08:52:03.113718 7f184e9f9e40  4 rocksdb:          Options.compression: NoCompression
126
2018-02-13 08:52:03.113720 7f184e9f9e40  4 rocksdb:                  Options.bottommost_compression: Disabled
127
2018-02-13 08:52:03.113721 7f184e9f9e40  4 rocksdb:       Options.prefix_extractor: nullptr
128
2018-02-13 08:52:03.113722 7f184e9f9e40  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
129
2018-02-13 08:52:03.113723 7f184e9f9e40  4 rocksdb:             Options.num_levels: 7
130
2018-02-13 08:52:03.113724 7f184e9f9e40  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
131
2018-02-13 08:52:03.113725 7f184e9f9e40  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
132
2018-02-13 08:52:03.113726 7f184e9f9e40  4 rocksdb:            Options.compression_opts.window_bits: -14
133
2018-02-13 08:52:03.113727 7f184e9f9e40  4 rocksdb:                  Options.compression_opts.level: -1
134
2018-02-13 08:52:03.113728 7f184e9f9e40  4 rocksdb:               Options.compression_opts.strategy: 0
135
2018-02-13 08:52:03.113728 7f184e9f9e40  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
136
2018-02-13 08:52:03.113729 7f184e9f9e40  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
137
2018-02-13 08:52:03.113730 7f184e9f9e40  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
138
2018-02-13 08:52:03.113731 7f184e9f9e40  4 rocksdb:              Options.level0_stop_writes_trigger: 36
139
2018-02-13 08:52:03.113732 7f184e9f9e40  4 rocksdb:                   Options.target_file_size_base: 67108864
140
2018-02-13 08:52:03.113733 7f184e9f9e40  4 rocksdb:             Options.target_file_size_multiplier: 1
141
2018-02-13 08:52:03.113733 7f184e9f9e40  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
142
2018-02-13 08:52:03.113734 7f184e9f9e40  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
143
2018-02-13 08:52:03.113735 7f184e9f9e40  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
144
2018-02-13 08:52:03.113741 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
145
2018-02-13 08:52:03.113742 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
146
2018-02-13 08:52:03.113744 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
147
2018-02-13 08:52:03.113745 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
148
2018-02-13 08:52:03.113746 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
149
2018-02-13 08:52:03.113747 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
150
2018-02-13 08:52:03.113748 7f184e9f9e40  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
151
2018-02-13 08:52:03.113750 7f184e9f9e40  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
152
2018-02-13 08:52:03.113751 7f184e9f9e40  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
153
2018-02-13 08:52:03.113752 7f184e9f9e40  4 rocksdb:                        Options.arena_block_size: 4194304
154
2018-02-13 08:52:03.113762 7f184e9f9e40  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
155
2018-02-13 08:52:03.113763 7f184e9f9e40  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
156
2018-02-13 08:52:03.113764 7f184e9f9e40  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
157
2018-02-13 08:52:03.113765 7f184e9f9e40  4 rocksdb:                Options.disable_auto_compactions: 0
158
2018-02-13 08:52:03.113766 7f184e9f9e40  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
159
2018-02-13 08:52:03.113767 7f184e9f9e40  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
160
2018-02-13 08:52:03.113768 7f184e9f9e40  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
161
2018-02-13 08:52:03.113769 7f184e9f9e40  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
162
2018-02-13 08:52:03.113770 7f184e9f9e40  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
163
2018-02-13 08:52:03.113771 7f184e9f9e40  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
164
2018-02-13 08:52:03.113772 7f184e9f9e40  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
165
2018-02-13 08:52:03.113773 7f184e9f9e40  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
166
2018-02-13 08:52:03.113774 7f184e9f9e40  4 rocksdb:                   Options.table_properties_collectors: 
167
2018-02-13 08:52:03.113775 7f184e9f9e40  4 rocksdb:                   Options.inplace_update_support: 0
168
2018-02-13 08:52:03.113775 7f184e9f9e40  4 rocksdb:                 Options.inplace_update_num_locks: 10000
169
2018-02-13 08:52:03.113776 7f184e9f9e40  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
170
2018-02-13 08:52:03.113778 7f184e9f9e40  4 rocksdb:   Options.memtable_huge_page_size: 0
171
2018-02-13 08:52:03.113779 7f184e9f9e40  4 rocksdb:                           Options.bloom_locality: 0
172
2018-02-13 08:52:03.113793 7f184e9f9e40  4 rocksdb:                    Options.max_successive_merges: 0
173
2018-02-13 08:52:03.113794 7f184e9f9e40  4 rocksdb:                Options.optimize_filters_for_hits: 0
174
2018-02-13 08:52:03.113795 7f184e9f9e40  4 rocksdb:                Options.paranoid_file_checks: 0
175
2018-02-13 08:52:03.113796 7f184e9f9e40  4 rocksdb:                Options.force_consistency_checks: 0
176
2018-02-13 08:52:03.113796 7f184e9f9e40  4 rocksdb:                Options.report_bg_io_stats: 0
177
2018-02-13 08:52:03.114800 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-smp-016/store.db/MANIFEST-000101 succeeded,manifest_file_number is 101, next_file_number is 103, last_sequence is 1427, log_number is 0,prev_log_number is 0,max_column_family is 0
178

    
179
2018-02-13 08:52:03.114810 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 100
180

    
181
2018-02-13 08:52:03.114894 7f184e9f9e40  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1518508323114872, "job": 1, "event": "recovery_started", "log_files": [102]}
182
2018-02-13 08:52:03.114904 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #102 mode 2
183
2018-02-13 08:52:03.133814 7f184e9f9e40  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1518508323133795, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 103, "file_size": 19004, "table_properties": {"data_size": 17923, "index_size": 87, "filter_size": 80, "raw_key_size": 330, "raw_average_key_size": 23, "raw_value_size": 17559, "raw_average_value_size": 1254, "num_data_blocks": 3, "num_entries": 14, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
184
2018-02-13 08:52:03.134044 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 104
185

    
186
2018-02-13 08:52:03.142897 7f184e9f9e40  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1518508323142892, "job": 1, "event": "recovery_finished"}
187
2018-02-13 08:52:03.155840 7f184e9f9e40  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5565f99a2000
188
2018-02-13 08:52:03.157072 7f184e9f9e40  1 RDMAStack RDMAStack ms_async_rdma_enable_hugepage value is: 0
189
2018-02-13 08:52:03.157096 7f184e9f9e40 20 RDMAStack RDMAStack constructing RDMAStack...
190
2018-02-13 08:52:03.157126 7f184e9f9e40 20 RDMAStack  creating RDMAStack:0x5565f99b4318 with dispatcher:0x5565f962a480
191
2018-02-13 08:52:03.157208 7f1840fac700  2 Event(0x5565f966ad00 nevent=5000 time_id=1).set_owner idx=0 owner=139742146119424
192
2018-02-13 08:52:03.157225 7f18417ad700  2 Event(0x5565f966b200 nevent=5000 time_id=1).set_owner idx=1 owner=139742154512128
193
2018-02-13 08:52:03.157234 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event started fd=16 mask=1 original mask is 0
194
2018-02-13 08:52:03.157239 7f1840fac700 20 EpollDriver.add_event add event fd=16 cur_mask=0 add_mask=1 to 15
195
2018-02-13 08:52:03.157242 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event started fd=19 mask=1 original mask is 0
196
2018-02-13 08:52:03.157238 7f1841fae700  2 Event(0x5565f966b700 nevent=5000 time_id=1).set_owner idx=2 owner=139742162904832
197
2018-02-13 08:52:03.157246 7f18417ad700 20 EpollDriver.add_event add event fd=19 cur_mask=0 add_mask=1 to 18
198
2018-02-13 08:52:03.157248 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event end fd=16 mask=1 original mask is 1
199
2018-02-13 08:52:03.157250 7f1840fac700 10 stack operator() starting
200
2018-02-13 08:52:03.157250 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event started fd=22 mask=1 original mask is 0
201
2018-02-13 08:52:03.157252 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event end fd=19 mask=1 original mask is 1
202
2018-02-13 08:52:03.157254 7f1841fae700 20 EpollDriver.add_event add event fd=22 cur_mask=0 add_mask=1 to 21
203
2018-02-13 08:52:03.157254 7f18417ad700 10 stack operator() starting
204
2018-02-13 08:52:03.157258 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event end fd=22 mask=1 original mask is 1
205
2018-02-13 08:52:03.157262 7f1841fae700 10 stack operator() starting
206
2018-02-13 08:52:03.157342 7f184e9f9e40  0 starting mon.smp-016 rank 0 at public addr 192.168.4.96:6789/0 at bind addr 192.168.4.96:6789/0 mon_data /var/lib/ceph/mon/ceph-smp-016 fsid 6de5466c-d11f-4e59-857b-b11cb0bc4d9b
207
2018-02-13 08:52:03.157381 7f184e9f9e40 10 -- - bind bind 192.168.4.96:6789/0
208
2018-02-13 08:52:03.157389 7f184e9f9e40 10 -- - bind Network Stack is not ready for bind yet - postponed
209
2018-02-13 08:52:03.157433 7f184e9f9e40  0 starting mon.smp-016 rank 0 at 192.168.4.96:6789/0 mon_data /var/lib/ceph/mon/ceph-smp-016 fsid 6de5466c-d11f-4e59-857b-b11cb0bc4d9b
210
2018-02-13 08:52:03.158216 7f184e9f9e40  1 mon.smp-016@-1(probing) e1 preinit fsid 6de5466c-d11f-4e59-857b-b11cb0bc4d9b
211
2018-02-13 08:52:03.159023 7f184e9f9e40  1 mon.smp-016@-1(probing).mds e0 Unable to load 'last_metadata'
212
2018-02-13 08:52:03.159151 7f184e9f9e40  1 mon.smp-016@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
213
2018-02-13 08:52:03.159162 7f184e9f9e40  1 mon.smp-016@-1(probing).pg v0 on_upgrade discarding in-core PGMap
214
2018-02-13 08:52:03.159379 7f184e9f9e40  0 mon.smp-016@-1(probing).mds e1 print_map
215
e1
216
enable_multiple, ever_enabled_multiple: 0,0
217
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
218
legacy client fscid: -1
219
 
220
No filesystems configured
221

    
222
2018-02-13 08:52:03.159612 7f184e9f9e40  0 mon.smp-016@-1(probing).osd e1 crush map has features 288514050185494528, adjusting msgr requires
223
2018-02-13 08:52:03.159621 7f184e9f9e40  0 mon.smp-016@-1(probing).osd e1 crush map has features 288514050185494528, adjusting msgr requires
224
2018-02-13 08:52:03.159629 7f184e9f9e40  0 mon.smp-016@-1(probing).osd e1 crush map has features 1009089990564790272, adjusting msgr requires
225
2018-02-13 08:52:03.159632 7f184e9f9e40  0 mon.smp-016@-1(probing).osd e1 crush map has features 288514050185494528, adjusting msgr requires
226
2018-02-13 08:52:03.160162 7f184e9f9e40  1 mon.smp-016@-1(probing).paxosservice(auth 1..157) refresh upgraded, format 0 -> 2
227
2018-02-13 08:52:03.164059 7f184e9f9e40  1 -- - start start
228
2018-02-13 08:52:03.164063 7f184e9f9e40  1 -- - start start
229
2018-02-13 08:52:03.164227 7f184e9f9e40 10 -- - ready -
230
2018-02-13 08:52:03.164232 7f184e9f9e40 10 -- - bind bind 192.168.4.96:6789/0
231
2018-02-13 08:52:03.164239 7f184e9f9e40 10  Processor -- bind
232
2018-02-13 08:52:03.164243 7f184e9f9e40 20 Event(0x5565f966ad00 nevent=5000 time_id=1).wakeup
233
2018-02-13 08:52:03.165805 7f1840fac700  1 Infiniband binding_port found active port 1
234
2018-02-13 08:52:03.165823 7f1840fac700  1 Infiniband init assigning: 1024 receive buffers
235
2018-02-13 08:52:03.165825 7f1840fac700  1 Infiniband init assigning: 1024 send buffers
236
2018-02-13 08:52:03.165828 7f1840fac700  1 Infiniband init device allow 4194303 completion entries
237
2018-02-13 08:52:03.229871 7f1840fac700 20 Infiniband init started.
238
2018-02-13 08:52:03.229910 7f1840fac700 20 Infiniband init started.
239
2018-02-13 08:52:03.231678 7f1840fac700 20 Infiniband init successfully create cq=0x5565f962a900
240
2018-02-13 08:52:03.233339 7f1840fac700 20 Infiniband init successfully create cq=0x5565f962aa80
241
2018-02-13 08:52:03.233471 7f1844d8f700 20 RDMAStack polling going to poll tx cq: 0x5565f99eb7d0 rx cq: 0x5565f99eb800
242
2018-02-13 08:52:03.233503 7f1844d8f700 20 Infiniband rearm_notify started.
243
2018-02-13 08:52:03.233505 7f1844d8f700 20 Infiniband rearm_notify started.
244
2018-02-13 08:52:03.234069 7f1840fac700 20  RDMAServerSocketImpl listen bind to 192.168.4.96:6789 on port 6789
245
2018-02-13 08:52:03.234167 7f184e9f9e40 10  Processor -- bind bound to 192.168.4.96:6789/0
246
2018-02-13 08:52:03.234185 7f184e9f9e40  1 -- 192.168.4.96:6789/0 learned_addr learned my addr 192.168.4.96:6789/0
247
2018-02-13 08:52:03.234190 7f184e9f9e40  1 -- 192.168.4.96:6789/0 _finish_bind bind my_inst.addr is 192.168.4.96:6789/0
248
2018-02-13 08:52:03.234197 7f184e9f9e40  1  Processor -- start
249
2018-02-13 08:52:03.234199 7f184e9f9e40 20 Event(0x5565f966ad00 nevent=5000 time_id=1).wakeup
250
2018-02-13 08:52:03.234217 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event started fd=28 mask=1 original mask is 0
251
2018-02-13 08:52:03.234223 7f1840fac700 20 EpollDriver.add_event add event fd=28 cur_mask=0 add_mask=1 to 15
252
2018-02-13 08:52:03.234229 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event end fd=28 mask=1 original mask is 1
253
2018-02-13 08:52:03.234333 7f184e9f9e40 10 -- - ready -
254
2018-02-13 08:52:03.234339 7f184e9f9e40  1  Processor -- start
255
2018-02-13 08:52:03.234385 7f184e9f9e40  0 mon.smp-016@-1(probing) e1  my rank is now 0 (was -1)
256
2018-02-13 08:52:03.234393 7f184e9f9e40  1 -- 192.168.4.96:6789/0 shutdown_connections 
257
2018-02-13 08:52:03.234431 7f184e9f9e40  1 mon.smp-016@0(probing) e1 win_standalone_election
258
2018-02-13 08:52:03.234465 7f184e9f9e40  1 mon.smp-016@0(probing).elector(37) init, last seen epoch 37, mid-election, bumping
259
2018-02-13 08:52:03.243410 7f184e9f9e40  0 log_channel(cluster) log [INF] : mon.smp-016@0 won leader election with quorum 0
260
2018-02-13 08:52:03.243498 7f184e9f9e40  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(1 entries from seq 1 at 2018-02-13 08:52:03.243423) v1 -- 0x5565f95e81c0 con 0
261
2018-02-13 08:52:03.243509 7f184e9f9e40 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(1 entries from seq 1 at 2018-02-13 08:52:03.243423) v1 local
262
2018-02-13 08:52:03.243560 7f184e9f9e40  0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {smp-016=192.168.4.96:6789/0}
263
2018-02-13 08:52:03.243580 7f184e9f9e40  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(1 entries from seq 2 at 2018-02-13 08:52:03.243565) v1 -- 0x5565f95e8400 con 0
264
2018-02-13 08:52:03.243576 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f95e81c0 prio 196
265
2018-02-13 08:52:03.243589 7f184e9f9e40 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(1 entries from seq 2 at 2018-02-13 08:52:03.243565) v1 local
266
2018-02-13 08:52:03.243608 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f95e8400 prio 196
267
2018-02-13 08:52:03.243638 7f184e9f9e40  0 log_channel(cluster) log [DBG] : fsmap 
268
2018-02-13 08:52:03.243661 7f184e9f9e40  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(1 entries from seq 3 at 2018-02-13 08:52:03.243643) v1 -- 0x5565f95e8640 con 0
269
2018-02-13 08:52:03.243672 7f184e9f9e40 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(1 entries from seq 3 at 2018-02-13 08:52:03.243643) v1 local
270
2018-02-13 08:52:03.243689 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f95e8640 prio 196
271
2018-02-13 08:52:03.243668 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(1 entries from seq 1 at 2018-02-13 08:52:03.243423) v1 ==== 0+0+0 (0 0 0) 0x5565f95e81c0 con 0x5565f9eb8000
272
2018-02-13 08:52:03.243743 7f184e9f9e40  0 log_channel(cluster) log [DBG] : osdmap e1: 0 total, 0 up, 0 in
273
2018-02-13 08:52:03.243775 7f184e9f9e40  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(1 entries from seq 4 at 2018-02-13 08:52:03.243748) v1 -- 0x5565f95e8880 con 0
274
2018-02-13 08:52:03.243784 7f184e9f9e40 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(1 entries from seq 4 at 2018-02-13 08:52:03.243748) v1 local
275
2018-02-13 08:52:03.243801 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f95e8880 prio 196
276
2018-02-13 08:52:03.243837 7f184e9f9e40  0 log_channel(cluster) log [DBG] : mgrmap e1: no daemons active
277
2018-02-13 08:52:03.243858 7f184e9f9e40  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(1 entries from seq 5 at 2018-02-13 08:52:03.243842) v1 -- 0x5565f95e8ac0 con 0
278
2018-02-13 08:52:03.243867 7f184e9f9e40 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(1 entries from seq 5 at 2018-02-13 08:52:03.243842) v1 local
279
2018-02-13 08:52:03.243881 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f95e8ac0 prio 196
280
2018-02-13 08:52:03.244177 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f95e81c0
281
2018-02-13 08:52:03.244193 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(1 entries from seq 2 at 2018-02-13 08:52:03.243565) v1 ==== 0+0+0 (0 0 0) 0x5565f95e8400 con 0x5565f9eb8000
282
2018-02-13 08:52:03.244235 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f95e8400
283
2018-02-13 08:52:03.244245 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(1 entries from seq 3 at 2018-02-13 08:52:03.243643) v1 ==== 0+0+0 (0 0 0) 0x5565f95e8640 con 0x5565f9eb8000
284
2018-02-13 08:52:03.244280 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f95e8640
285
2018-02-13 08:52:03.244289 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(1 entries from seq 4 at 2018-02-13 08:52:03.243748) v1 ==== 0+0+0 (0 0 0) 0x5565f95e8880 con 0x5565f9eb8000
286
2018-02-13 08:52:03.244337 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f95e8880
287
2018-02-13 08:52:03.244350 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(1 entries from seq 5 at 2018-02-13 08:52:03.243842) v1 ==== 0+0+0 (0 0 0) 0x5565f95e8ac0 con 0x5565f9eb8000
288
2018-02-13 08:52:03.244390 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f95e8ac0
289
2018-02-13 08:52:03.304374 7f18407ab700  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(last 1) v1 -- 0x5565f9a80b20 con 0
290
2018-02-13 08:52:03.304391 7f18407ab700 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(last 1) v1 local
291
2018-02-13 08:52:03.304435 7f18407ab700  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(last 2) v1 -- 0x5565f9a7ffe0 con 0
292
2018-02-13 08:52:03.304472 7f18407ab700 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(last 2) v1 local
293
2018-02-13 08:52:03.304478 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f9a80b20 prio 196
294
2018-02-13 08:52:03.304515 7f18407ab700  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(last 3) v1 -- 0x5565f9a801c0 con 0
295
2018-02-13 08:52:03.304532 7f18407ab700 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(last 3) v1 local
296
2018-02-13 08:52:03.304535 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f9a7ffe0 prio 196
297
2018-02-13 08:52:03.304543 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f9a801c0 prio 196
298
2018-02-13 08:52:03.304550 7f18407ab700  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(last 4) v1 -- 0x5565f9a803a0 con 0
299
2018-02-13 08:52:03.304555 7f18407ab700 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(last 4) v1 local
300
2018-02-13 08:52:03.304577 7f18407ab700  1 -- 192.168.4.96:6789/0 --> 192.168.4.96:6789/0 -- log(last 5) v1 -- 0x5565f9a7e1e0 con 0
301
2018-02-13 08:52:03.304580 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f9a803a0 prio 196
302
2018-02-13 08:52:03.304584 7f18407ab700 20 -- 192.168.4.96:6789/0 >> 192.168.4.96:6789/0 conn(0x5565f9eb8000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).send_message log(last 5) v1 local
303
2018-02-13 08:52:03.304578 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x5565f9a80b20 con 0x5565f9eb8000
304
2018-02-13 08:52:03.304597 7f1843d8d700 20 -- 192.168.4.96:6789/0 queue 0x5565f9a7e1e0 prio 196
305
2018-02-13 08:52:03.304643 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f9a80b20
306
2018-02-13 08:52:03.304654 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x5565f9a7ffe0 con 0x5565f9eb8000
307
2018-02-13 08:52:03.304681 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f9a7ffe0
308
2018-02-13 08:52:03.304689 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x5565f9a801c0 con 0x5565f9eb8000
309
2018-02-13 08:52:03.304715 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f9a801c0
310
2018-02-13 08:52:03.304736 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x5565f9a803a0 con 0x5565f9eb8000
311
2018-02-13 08:52:03.304770 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f9a803a0
312
2018-02-13 08:52:03.304781 7f184458e700  1 -- 192.168.4.96:6789/0 <== mon.0 192.168.4.96:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x5565f9a7e1e0 con 0x5565f9eb8000
313
2018-02-13 08:52:03.304818 7f184458e700 20 -- 192.168.4.96:6789/0 done calling dispatch on 0x5565f9a7e1e0
314
2018-02-13 08:52:17.255580 7f1840fac700 10  Processor -- accept listen_fd=28
315
2018-02-13 08:52:17.255613 7f1840fac700 15  RDMAServerSocketImpl accept
316
2018-02-13 08:52:17.255652 7f1840fac700 20 Infiniband init started.
317
2018-02-13 08:52:17.255775 7f1840fac700 20 Infiniband init successfully create queue pair: qp=0x5565f966ca80
318
2018-02-13 08:52:17.256206 7f1840fac700 20 Infiniband init successfully change queue pair to INIT: qp=0x5565f966ca80
319
2018-02-13 08:52:17.256229 7f1840fac700 20 Event(0x5565f966b700 nevent=5000 time_id=1).wakeup
320
2018-02-13 08:52:17.256241 7f1840fac700 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 37
321
2018-02-13 08:52:17.256244 7f1840fac700 10  Processor -- accept accepted incoming on sd 38
322
2018-02-13 08:52:17.256283 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=38
323
2018-02-13 08:52:17.256296 7f1840fac700 20 Event(0x5565f966b700 nevent=5000 time_id=1).wakeup
324
2018-02-13 08:52:17.256291 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event started fd=37 mask=1 original mask is 0
325
2018-02-13 08:52:17.256300 7f1841fae700 20 EpollDriver.add_event add event fd=37 cur_mask=0 add_mask=1 to 21
326
2018-02-13 08:52:17.256301 7f1840fac700 15  RDMAServerSocketImpl accept
327
2018-02-13 08:52:17.256306 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event end fd=37 mask=1 original mask is 1
328
2018-02-13 08:52:17.256320 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262718 tcp_fd: 37 notify_fd: 38
329
2018-02-13 08:52:17.256342 7f1841fae700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???A
330
2018-02-13 08:52:17.256351 7f1841fae700 10 Infiniband send_msg sending: 79, 262718, 0, 0, fe80000000000000f45214030035f771
331
2018-02-13 08:52:17.256394 7f1841fae700 20  RDMAConnectedSocketImpl activate Choosing gid_index 67, sl 3
332
2018-02-13 08:52:17.256456 7f1841fae700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
333
2018-02-13 08:52:17.256513 7f1841fae700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
334
2018-02-13 08:52:17.256517 7f1841fae700 20  RDMAConnectedSocketImpl activate QueuePair: 0x5565f9ec3030 with qp:0x5565f966ca80
335
2018-02-13 08:52:17.256525 7f1841fae700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :-1 s=STATE_ACCEPTING pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
336
2018-02-13 08:52:17.256555 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event started fd=38 mask=1 original mask is 0
337
2018-02-13 08:52:17.256559 7f1841fae700 20 EpollDriver.add_event add event fd=38 cur_mask=0 add_mask=1 to 21
338
2018-02-13 08:52:17.256562 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event end fd=38 mask=1 original mask is 1
339
2018-02-13 08:52:17.256572 7f1841fae700  1 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=38 -
340
2018-02-13 08:52:17.256581 7f1841fae700 20  RDMAConnectedSocketImpl send fake send to upper, QP: 262718
341
2018-02-13 08:52:17.256583 7f1841fae700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
342
2018-02-13 08:52:17.256590 7f1841fae700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
343
2018-02-13 08:52:17.256611 7f1841fae700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aac000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
344
2018-02-13 08:52:17.256620 7f1841fae700 20  RDMAConnectedSocketImpl read notify_fd : 0 in 262718 r = -1
345
2018-02-13 08:52:17.256628 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262718 tcp_fd: 37 notify_fd: 38
346
2018-02-13 08:52:17.256637 7f1841fae700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???A
347
2018-02-13 08:52:17.256641 7f1841fae700 10  RDMAConnectedSocketImpl handle_connection server is already active.
348
2018-02-13 08:52:21.498567 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262718 tcp_fd: 37 notify_fd: 38
349
2018-02-13 08:52:21.498596 7f1841fae700 10 Infiniband recv_msg got disconnect message 
350
2018-02-13 08:52:21.498599 7f1841fae700 10  RDMAConnectedSocketImpl handle_connection server is already active.
351
2018-02-13 08:52:21.699958 7f1840fac700 10  Processor -- accept listen_fd=28
352
2018-02-13 08:52:21.699980 7f1840fac700 15  RDMAServerSocketImpl accept
353
2018-02-13 08:52:21.700009 7f1840fac700 20 Infiniband init started.
354
2018-02-13 08:52:21.700179 7f1840fac700 20 Infiniband init successfully create queue pair: qp=0x5565f966cd00
355
2018-02-13 08:52:21.700843 7f1840fac700 20 Infiniband init successfully change queue pair to INIT: qp=0x5565f966cd00
356
2018-02-13 08:52:21.700863 7f1840fac700 20 Event(0x5565f966b200 nevent=5000 time_id=1).wakeup
357
2018-02-13 08:52:21.700879 7f1840fac700 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 39
358
2018-02-13 08:52:21.700883 7f1840fac700 10  Processor -- accept accepted incoming on sd 40
359
2018-02-13 08:52:21.700903 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=40
360
2018-02-13 08:52:21.700919 7f1840fac700 15  RDMAServerSocketImpl accept
361
2018-02-13 08:52:21.700937 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event started fd=39 mask=1 original mask is 0
362
2018-02-13 08:52:21.700950 7f18417ad700 20 EpollDriver.add_event add event fd=39 cur_mask=0 add_mask=1 to 18
363
2018-02-13 08:52:21.700959 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event end fd=39 mask=1 original mask is 1
364
2018-02-13 08:52:21.700965 7f18417ad700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :-1 s=STATE_ACCEPTING pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
365
2018-02-13 08:52:21.701001 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event started fd=40 mask=1 original mask is 0
366
2018-02-13 08:52:21.701006 7f18417ad700 20 EpollDriver.add_event add event fd=40 cur_mask=0 add_mask=1 to 18
367
2018-02-13 08:52:21.701012 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event end fd=40 mask=1 original mask is 1
368
2018-02-13 08:52:21.701043 7f18417ad700  1 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=40 -
369
2018-02-13 08:52:21.701058 7f18417ad700 20  RDMAConnectedSocketImpl send fake send to upper, QP: 262720
370
2018-02-13 08:52:21.701062 7f18417ad700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
371
2018-02-13 08:52:21.701071 7f18417ad700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
372
2018-02-13 08:52:21.701077 7f18417ad700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aad800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
373
2018-02-13 08:52:21.701087 7f18417ad700 20  RDMAConnectedSocketImpl read notify_fd : 0 in 262720 r = -1
374
2018-02-13 08:52:21.701098 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262720 tcp_fd: 39 notify_fd: 40
375
2018-02-13 08:52:21.701117 7f18417ad700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ??zA
376
2018-02-13 08:52:21.701131 7f18417ad700 10 Infiniband send_msg sending: 79, 262720, 2116118, 0, fe80000000000000f45214030035f771
377
2018-02-13 08:52:21.701203 7f18417ad700 20  RDMAConnectedSocketImpl activate Choosing gid_index 67, sl 3
378
2018-02-13 08:52:21.701300 7f18417ad700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
379
2018-02-13 08:52:21.701414 7f18417ad700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
380
2018-02-13 08:52:21.701422 7f18417ad700 20  RDMAConnectedSocketImpl activate QueuePair: 0x5565f9ec3110 with qp:0x5565f966cd00
381
2018-02-13 08:52:21.701573 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262720 tcp_fd: 39 notify_fd: 40
382
2018-02-13 08:52:21.701593 7f18417ad700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ??zA
383
2018-02-13 08:52:21.701597 7f18417ad700 10  RDMAConnectedSocketImpl handle_connection server is already active.
384
2018-02-13 08:52:25.827257 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262720 tcp_fd: 39 notify_fd: 40
385
2018-02-13 08:52:25.827286 7f18417ad700 10 Infiniband recv_msg got disconnect message 
386
2018-02-13 08:52:25.827289 7f18417ad700 10  RDMAConnectedSocketImpl handle_connection server is already active.
387
2018-02-13 08:52:26.228799 7f1840fac700 10  Processor -- accept listen_fd=28
388
2018-02-13 08:52:26.228821 7f1840fac700 15  RDMAServerSocketImpl accept
389
2018-02-13 08:52:26.228850 7f1840fac700 20 Infiniband init started.
390
2018-02-13 08:52:26.229014 7f1840fac700 20 Infiniband init successfully create queue pair: qp=0x5565f966cf80
391
2018-02-13 08:52:26.229480 7f1840fac700 20 Infiniband init successfully change queue pair to INIT: qp=0x5565f966cf80
392
2018-02-13 08:52:26.229501 7f1840fac700 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 41
393
2018-02-13 08:52:26.229505 7f1840fac700 10  Processor -- accept accepted incoming on sd 42
394
2018-02-13 08:52:26.229530 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=42
395
2018-02-13 08:52:26.229552 7f1840fac700 15  RDMAServerSocketImpl accept
396
2018-02-13 08:52:26.229573 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event started fd=41 mask=1 original mask is 0
397
2018-02-13 08:52:26.229581 7f1840fac700 20 EpollDriver.add_event add event fd=41 cur_mask=0 add_mask=1 to 15
398
2018-02-13 08:52:26.229590 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event end fd=41 mask=1 original mask is 1
399
2018-02-13 08:52:26.229597 7f1840fac700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :-1 s=STATE_ACCEPTING pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
400
2018-02-13 08:52:26.229606 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event started fd=42 mask=1 original mask is 0
401
2018-02-13 08:52:26.229609 7f1840fac700 20 EpollDriver.add_event add event fd=42 cur_mask=0 add_mask=1 to 15
402
2018-02-13 08:52:26.229615 7f1840fac700 20 Event(0x5565f966ad00 nevent=5000 time_id=1).create_file_event create event end fd=42 mask=1 original mask is 1
403
2018-02-13 08:52:26.229624 7f1840fac700  1 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=42 -
404
2018-02-13 08:52:26.229638 7f1840fac700 20  RDMAConnectedSocketImpl send fake send to upper, QP: 262722
405
2018-02-13 08:52:26.229640 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
406
2018-02-13 08:52:26.229647 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
407
2018-02-13 08:52:26.229654 7f1840fac700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9aaf000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
408
2018-02-13 08:52:26.229664 7f1840fac700 20  RDMAConnectedSocketImpl read notify_fd : 0 in 262722 r = -1
409
2018-02-13 08:52:26.229675 7f1840fac700 20  RDMAConnectedSocketImpl handle_connection QP: 262722 tcp_fd: 41 notify_fd: 42
410
2018-02-13 08:52:26.229716 7f1840fac700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???@
411
2018-02-13 08:52:26.229732 7f1840fac700 10 Infiniband send_msg sending: 79, 262722, 5515815, 0, fe80000000000000f45214030035f771
412
2018-02-13 08:52:26.229802 7f1840fac700 20  RDMAConnectedSocketImpl activate Choosing gid_index 67, sl 3
413
2018-02-13 08:52:26.229866 7f1840fac700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
414
2018-02-13 08:52:26.229919 7f1840fac700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
415
2018-02-13 08:52:26.229926 7f1840fac700 20  RDMAConnectedSocketImpl activate QueuePair: 0x5565f9ec2690 with qp:0x5565f966cf80
416
2018-02-13 08:52:26.230103 7f1840fac700 20  RDMAConnectedSocketImpl handle_connection QP: 262722 tcp_fd: 41 notify_fd: 42
417
2018-02-13 08:52:26.230121 7f1840fac700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???@
418
2018-02-13 08:52:26.230128 7f1840fac700 10  RDMAConnectedSocketImpl handle_connection server is already active.
419
2018-02-13 08:52:27.176569 7f1840fac700 20  RDMAConnectedSocketImpl handle_connection QP: 262722 tcp_fd: 41 notify_fd: 42
420
2018-02-13 08:52:27.176598 7f1840fac700 10 Infiniband recv_msg got disconnect message 
421
2018-02-13 08:52:27.176600 7f1840fac700 10  RDMAConnectedSocketImpl handle_connection server is already active.
422
2018-02-13 08:52:27.177517 7f1840fac700 10  Processor -- accept listen_fd=28
423
2018-02-13 08:52:27.177540 7f1840fac700 15  RDMAServerSocketImpl accept
424
2018-02-13 08:52:27.177571 7f1840fac700 20 Infiniband init started.
425
2018-02-13 08:52:27.177735 7f1840fac700 20 Infiniband init successfully create queue pair: qp=0x5565f966d200
426
2018-02-13 08:52:27.178243 7f1840fac700 20 Infiniband init successfully change queue pair to INIT: qp=0x5565f966d200
427
2018-02-13 08:52:27.178270 7f1840fac700 20 Event(0x5565f966b700 nevent=5000 time_id=1).wakeup
428
2018-02-13 08:52:27.178283 7f1840fac700 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 43
429
2018-02-13 08:52:27.178286 7f1840fac700 10  Processor -- accept accepted incoming on sd 44
430
2018-02-13 08:52:27.178309 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=44
431
2018-02-13 08:52:27.178326 7f1840fac700 15  RDMAServerSocketImpl accept
432
2018-02-13 08:52:27.178368 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event started fd=43 mask=1 original mask is 0
433
2018-02-13 08:52:27.178390 7f1841fae700 20 EpollDriver.add_event add event fd=43 cur_mask=0 add_mask=1 to 21
434
2018-02-13 08:52:27.178399 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event end fd=43 mask=1 original mask is 1
435
2018-02-13 08:52:27.178408 7f1841fae700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :-1 s=STATE_ACCEPTING pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
436
2018-02-13 08:52:27.178426 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event started fd=44 mask=1 original mask is 0
437
2018-02-13 08:52:27.178433 7f1841fae700 20 EpollDriver.add_event add event fd=44 cur_mask=0 add_mask=1 to 21
438
2018-02-13 08:52:27.178465 7f1841fae700 20 Event(0x5565f966b700 nevent=5000 time_id=1).create_file_event create event end fd=44 mask=1 original mask is 1
439
2018-02-13 08:52:27.178494 7f1841fae700  1 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=44 -
440
2018-02-13 08:52:27.178509 7f1841fae700 20  RDMAConnectedSocketImpl send fake send to upper, QP: 262724
441
2018-02-13 08:52:27.178513 7f1841fae700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
442
2018-02-13 08:52:27.178520 7f1841fae700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
443
2018-02-13 08:52:27.178527 7f1841fae700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ab0800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
444
2018-02-13 08:52:27.178536 7f1841fae700 20  RDMAConnectedSocketImpl read notify_fd : 0 in 262724 r = -1
445
2018-02-13 08:52:27.178549 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262724 tcp_fd: 43 notify_fd: 44
446
2018-02-13 08:52:27.178570 7f1841fae700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???A
447
2018-02-13 08:52:27.178585 7f1841fae700 10 Infiniband send_msg sending: 79, 262724, 10238434, 0, fe80000000000000f45214030035f771
448
2018-02-13 08:52:27.178655 7f1841fae700 20  RDMAConnectedSocketImpl activate Choosing gid_index 67, sl 3
449
2018-02-13 08:52:27.178723 7f1841fae700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
450
2018-02-13 08:52:27.178805 7f1841fae700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
451
2018-02-13 08:52:27.178811 7f1841fae700 20  RDMAConnectedSocketImpl activate QueuePair: 0x5565f9ec2fc0 with qp:0x5565f966d200
452
2018-02-13 08:52:27.178903 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262724 tcp_fd: 43 notify_fd: 44
453
2018-02-13 08:52:27.178920 7f1841fae700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ???A
454
2018-02-13 08:52:27.178924 7f1841fae700 10  RDMAConnectedSocketImpl handle_connection server is already active.
455
2018-02-13 08:52:30.177084 7f1841fae700 20  RDMAConnectedSocketImpl handle_connection QP: 262724 tcp_fd: 43 notify_fd: 44
456
2018-02-13 08:52:30.177111 7f1841fae700 10 Infiniband recv_msg got disconnect message 
457
2018-02-13 08:52:30.177113 7f1841fae700 10  RDMAConnectedSocketImpl handle_connection server is already active.
458
2018-02-13 08:52:30.178099 7f1840fac700 10  Processor -- accept listen_fd=28
459
2018-02-13 08:52:30.178121 7f1840fac700 15  RDMAServerSocketImpl accept
460
2018-02-13 08:52:30.178152 7f1840fac700 20 Infiniband init started.
461
2018-02-13 08:52:30.178315 7f1840fac700 20 Infiniband init successfully create queue pair: qp=0x5565f966d480
462
2018-02-13 08:52:30.178833 7f1840fac700 20 Infiniband init successfully change queue pair to INIT: qp=0x5565f966d480
463
2018-02-13 08:52:30.178850 7f1840fac700 20 Event(0x5565f966b200 nevent=5000 time_id=1).wakeup
464
2018-02-13 08:52:30.178857 7f1840fac700 20  RDMAServerSocketImpl accept accepted a new QP, tcp_fd: 45
465
2018-02-13 08:52:30.178859 7f1840fac700 10  Processor -- accept accepted incoming on sd 46
466
2018-02-13 08:52:30.178887 7f1840fac700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=46
467
2018-02-13 08:52:30.178896 7f1840fac700 20 Event(0x5565f966b200 nevent=5000 time_id=1).wakeup
468
2018-02-13 08:52:30.178891 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event started fd=45 mask=1 original mask is 0
469
2018-02-13 08:52:30.178899 7f18417ad700 20 EpollDriver.add_event add event fd=45 cur_mask=0 add_mask=1 to 18
470
2018-02-13 08:52:30.178901 7f1840fac700 15  RDMAServerSocketImpl accept
471
2018-02-13 08:52:30.178904 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event end fd=45 mask=1 original mask is 1
472
2018-02-13 08:52:30.178912 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262726 tcp_fd: 45 notify_fd: 46
473
2018-02-13 08:52:30.178922 7f18417ad700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ??zA
474
2018-02-13 08:52:30.178934 7f18417ad700 10 Infiniband send_msg sending: 79, 262726, 11225430, 0, fe80000000000000f45214030035f771
475
2018-02-13 08:52:30.178974 7f18417ad700 20  RDMAConnectedSocketImpl activate Choosing gid_index 67, sl 3
476
2018-02-13 08:52:30.179026 7f18417ad700 20  RDMAConnectedSocketImpl activate transition to RTR state successfully.
477
2018-02-13 08:52:30.179099 7f18417ad700 20  RDMAConnectedSocketImpl activate transition to RTS state successfully.
478
2018-02-13 08:52:30.179103 7f18417ad700 20  RDMAConnectedSocketImpl activate QueuePair: 0x5565f9ec2f50 with qp:0x5565f966d480
479
2018-02-13 08:52:30.179107 7f18417ad700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :-1 s=STATE_ACCEPTING pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
480
2018-02-13 08:52:30.179115 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event started fd=46 mask=1 original mask is 0
481
2018-02-13 08:52:30.179117 7f18417ad700 20 EpollDriver.add_event add event fd=46 cur_mask=0 add_mask=1 to 18
482
2018-02-13 08:52:30.179120 7f18417ad700 20 Event(0x5565f966b200 nevent=5000 time_id=1).create_file_event create event end fd=46 mask=1 original mask is 1
483
2018-02-13 08:52:30.179126 7f18417ad700  1 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=46 -
484
2018-02-13 08:52:30.179134 7f18417ad700 20  RDMAConnectedSocketImpl send fake send to upper, QP: 262726
485
2018-02-13 08:52:30.179135 7f18417ad700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
486
2018-02-13 08:52:30.179140 7f18417ad700 10 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
487
2018-02-13 08:52:30.179145 7f18417ad700 20 -- 192.168.4.96:6789/0 >> - conn(0x5565f9ad6000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).process prev state is STATE_ACCEPTING
488
2018-02-13 08:52:30.179164 7f18417ad700 20  RDMAConnectedSocketImpl read notify_fd : 0 in 262726 r = -1
489
2018-02-13 08:52:30.179215 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262726 tcp_fd: 45 notify_fd: 46
490
2018-02-13 08:52:30.179234 7f18417ad700  5 Infiniband recv_msg recevd: 4, 0, 0, 0, ??zA
491
2018-02-13 08:52:30.179243 7f18417ad700 10  RDMAConnectedSocketImpl handle_connection server is already active.
492
2018-02-13 08:52:30.457113 7f18417ad700 20  RDMAConnectedSocketImpl handle_connection QP: 262726 tcp_fd: 45 notify_fd: 46
493
2018-02-13 08:52:30.457137 7f18417ad700 10 Infiniband recv_msg got disconnect message 
494
2018-02-13 08:52:30.457139 7f18417ad700 10  RDMAConnectedSocketImpl handle_connection server is already active.
495
2018-02-13 08:53:03.163979 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58791 MB
496
2018-02-13 08:54:03.164224 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB
497
2018-02-13 08:55:03.164517 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB
498
2018-02-13 08:56:03.164748 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB
499
2018-02-13 08:57:03.164970 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB
500
2018-02-13 08:58:03.165199 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB
501
2018-02-13 08:59:03.165434 7f18477b9700  0 mon.smp-016@0(leader).data_health(39) update_stats avail 97% total 60510 MB, used 1718 MB, avail 58792 MB