Ceph

Hi,

I have created a service with ETCD KV backend:

docker service create -d --network=host \
--mount type=bind,source=/var/lib/ceph,destination=/var/lib/ceph \
--mount type=bind,source=/var/log/ceph,destination=/var/log/ceph \
--mount type=bind,source=/etc/localtime,destination=/etc/localtime,readonly \
-e NETWORK_AUTO_DETECT=4 \
-e CEPH_NIC=interconnect \
-e KV_TYPE=etcd \
--secret=vdicube_etcd_passwd \
--name=vdicube_ceph_mon \
--mode=global \
--constraint node.labels.mon==1 \
vdicube/ceph_daemon:nautilus mon

--> Note, I have made a small update to get the IP address from an specific NIC and acceed etcd with a password stored in a swarm secret.

The first time I start the swarm service (in three nodes: vdicnode04, vdicnode05 and vdicnode06) it starts perfectly but when I stop and start the service again it just starts the first node vdicnode04.

Log of vdicnode05 (vdicnode06 log is the same)

Secret found. Retrieving etcd password from secret,
2019-09-05 23:09:11 /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster...,
2019-09-05 23:09:12 /opt/ceph-container/bin/entrypoint.sh: 'auth' key already exists,
2019-09-05 23:09:12 /opt/ceph-container/bin/entrypoint.sh: 'global' key already exists,
2019-09-05 23:09:12 /opt/ceph-container/bin/entrypoint.sh: 'mon' key already exists,
2019-09-05 23:09:13 /opt/ceph-container/bin/entrypoint.sh: 'mds' key already exists,
2019-09-05 23:09:13 /opt/ceph-container/bin/entrypoint.sh: 'osd' key already exists,
2019-09-05 23:09:13 /opt/ceph-container/bin/entrypoint.sh: 'client' key already exists,
2019-09-05 23:09:14 /opt/ceph-container/bin/entrypoint.sh: 'client_host' key already exists,
2019-09-05 23:09:14 /opt/ceph-container/bin/entrypoint.sh: Adding Mon Host - vdicnode05.,
192.168.100.105,
vdicnode05,
true,
2019-09-05 23:09:15 /opt/ceph-container/bin/entrypoint.sh: Configuration found for cluster ceph. Writing to disk.,
true,
2019-09-05T23:09:16+02:00 vdicnode05 /tmp/tmp.e8bjQ9FhaB/confd³⁴³: INFO Backend set to etcd,
2019-09-05T23:09:16+02:00 vdicnode05 /tmp/tmp.e8bjQ9FhaB/confd³⁴³: INFO Starting confd,
2019-09-05T23:09:16+02:00 vdicnode05 /tmp/tmp.e8bjQ9FhaB/confd³⁴³: INFO Backend source(s) set to http://127.0.0.1:2379,
2019-09-05T23:09:16+02:00 vdicnode05 /tmp/tmp.e8bjQ9FhaB/confd³⁴³: INFO Target config /etc/ceph/ceph.conf out of sync,
2019-09-05T23:09:16+02:00 vdicnode05 /tmp/tmp.e8bjQ9FhaB/confd³⁴³: INFO Target config /etc/ceph/ceph.conf has been updated,
2019-09-05 23:09:16 /opt/ceph-container/bin/entrypoint.sh: Adding bootstrap keyrings.,
2019-09-05 23:09:17 /opt/ceph-container/bin/entrypoint.sh: Adding mon/admin Keyrings.,
2019-09-05 23:09:18 /opt/ceph-container/bin/entrypoint.sh: Monmap is missing. Adding initial monmap...,
2019-09-05 23:09:18 /opt/ceph-container/bin/entrypoint.sh: Trying to get the most recent monmap...,
2019-09-05 23:09:23 /opt/ceph-container/bin/entrypoint.sh: Peers not found, using initial monmap.,
2019-09-05 23:09:23 /opt/ceph-container/bin/entrypoint.sh: Removing lock for vdicnode05.,
PrevNode.Value: vdicnode05,
2019-09-05 23:09:34 /opt/ceph-container/bin/entrypoint.sh: Adding monitor: vdicnode05 192.168.100.105:6789,
2019-09-05 23:09:41 /opt/ceph-container/bin/entrypoint.sh: Found 324MB of available memory (340381696 bytes),
2019-09-05 23:09:41 /opt/ceph-container/bin/entrypoint.sh: Tuning memory : osd_memory_base=162MB, osd_memory_cache_min=218MB, osd_memory_target=274MB,
2019-09-05 23:09:41 /opt/ceph-container/bin/entrypoint.sh: SUCCESS,
exec: PID 631: spawning /usr/bin/ceph-mon --cluster ceph --default-log-to-file=false --default-mon-cluster-log-to-file=false --setuser ceph --setgroup ceph d --mon-cluster-log-to-stderr --log-stderr-prefix=debug -i vdicnode05 --mon-data /var/lib/ceph/mon/ceph-vdicnode05 --public-addr 192.168.100.105,
exec: Waiting 631 to quit,
debug 2019-09-05 23:09:41.726 7f58630cc200 0 set uid:gid to 167:167 (ceph:ceph),
debug 2019-09-05 23:09:41.727 7f58630cc200 0 ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable), process ceph-mon, pid 631,
debug 2019-09-05 23:09:41.727 7f58630cc200 0 pidfile_write: ignore empty --pid-file,
debug 2019-09-05 23:09:41.831 7f58630cc200 0 load: jerasure load: lrc load: isa ,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option compression = kNoCompression,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option level_compaction_dynamic_level_bytes = true,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option write_buffer_size = 33554432,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option compression = kNoCompression,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option level_compaction_dynamic_level_bytes = true,
debug 2019-09-05 23:09:41.832 7f58630cc200 0 set rocksdb option write_buffer_size = 33554432,
debug 2019-09-05 23:09:41.832 7f58630cc200 1 rocksdb: do_open column families: [default],
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: RocksDB version: 6.1.2,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Git sha rocksdb_build_git_sha:0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Compile date Sep 3 2019,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: DB SUMMARY,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: CURRENT file: CURRENT,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: IDENTITY file: IDENTITY,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: MANIFEST file: MANIFEST-000122 size: 254 Bytes,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-vdicnode05/store.db dir, Total Num: 4, files: 000102.sst 000106.sst 000112.sst 000118.sst ,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-vdicnode05/store.db: 000123.log size: 431 ; ,
,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.error_if_exists: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.create_if_missing: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.paranoid_checks: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.env: 0x556b871fd600,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.info_log: 0x556b889a6320,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_file_opening_threads: 16,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.statistics: (nil),
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.use_fsync: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_log_file_size: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_manifest_file_size: 1073741824,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.log_file_time_to_roll: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.keep_log_file_num: 1000,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.recycle_log_file_num: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.allow_fallocate: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.allow_mmap_reads: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.allow_mmap_writes: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.use_direct_reads: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.create_missing_column_families: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.db_log_dir: ,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-vdicnode05/store.db,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.table_cache_numshardbits: 6,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_subcompactions: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_background_flushes: -1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.WAL_ttl_seconds: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.WAL_size_limit_MB: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.manifest_preallocation_size: 4194304,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.is_fd_close_on_exec: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.advise_random_on_open: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.db_write_buffer_size: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.write_buffer_manager: 0x556b88a620f0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.access_hint_on_compaction_start: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.random_access_max_buffer_size: 1048576,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.use_adaptive_mutex: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.rate_limiter: (nil),
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.wal_recovery_mode: 2,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.enable_thread_tracking: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.enable_pipelined_write: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.allow_concurrent_memtable_write: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.write_thread_max_yield_usec: 100,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.write_thread_slow_yield_usec: 3,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.row_cache: None,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.wal_filter: None,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.avoid_flush_during_recovery: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.allow_ingest_behind: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.preserve_deletes: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.two_write_queues: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.manual_wal_flush: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.atomic_flush: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_background_jobs: 2,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_background_compactions: -1,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.avoid_flush_during_shutdown: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.writable_file_max_buffer_size: 1048576,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.delayed_write_rate : 16777216,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.max_total_wal_size: 0,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.stats_dump_period_sec: 600,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.stats_persist_period_sec: 600,
debug 2019-09-05 23:09:41.832 7f58630cc200 4 rocksdb: Options.stats_history_buffer_size: 1048576,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.max_open_files: -1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bytes_per_sync: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.wal_bytes_per_sync: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compaction_readahead_size: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Compression algorithms supported:,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kZSTDNotFinalCompression supported: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kZSTD supported: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kXpressCompression supported: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kLZ4HCCompression supported: 1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kLZ4Compression supported: 1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kBZip2Compression supported: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kZlibCompression supported: 1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: kSnappyCompression supported: 1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Fast CRC32 supported: Supported on x86,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-vdicnode05/store.db/MANIFEST-000122,
,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: [db/column_family.cc:477] -------------- Options for column family [default]:,
,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.comparator: leveldb.BytewiseComparator,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.merge_operator: ,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compaction_filter: None,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compaction_filter_factory: None,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.memtable_factory: SkipListFactory,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.table_factory: BlockBasedTable,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b87cfa2e8),
cache_index_and_filter_blocks: 1,
cache_index_and_filter_blocks_with_high_priority: 1,
pin_l0_filter_and_index_blocks_in_cache: 1,
pin_top_level_index_and_filter: 1,
index_type: 0,
data_block_index_type: 0,
data_block_hash_table_util_ratio: 0.750000,
hash_index_allow_collision: 1,
checksum: 1,
no_block_cache: 0,
block_cache: 0x556b87d33390,
block_cache_name: BinnedLRUCache,
block_cache_options:,
capacity : 536870912,
num_shard_bits : 4,
strict_capacity_limit : 0,
high_pri_pool_ratio: 0.000,
block_cache_compressed: (nil),
persistent_cache: (nil),
block_size: 4096,
block_size_deviation: 10,
block_restart_interval: 16,
index_block_restart_interval: 1,
metadata_block_size: 4096,
partition_filters: 0,
use_delta_encoding: 1,
filter_policy: rocksdb.BuiltinBloomFilter,
whole_key_filtering: 1,
verify_compression: 0,
read_amp_bytes_per_bit: 0,
format_version: 2,
enable_index_compression: 1,
block_align: 0,
,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.write_buffer_size: 33554432,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.max_write_buffer_number: 2,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression: NoCompression,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression: Disabled,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.prefix_extractor: nullptr,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.num_levels: 7,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.min_write_buffer_number_to_merge: 1,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.window_bits: 14,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.level: 32767,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.strategy: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.bottommost_compression_opts.enabled: false,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.window_bits: -14,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.level: 32767,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.strategy: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.max_dict_bytes: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.compression_opts.enabled: false,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.level0_file_num_compaction_trigger: 4,
debug 2019-09-05 23:09:41.839 7f58630cc200 4 rocksdb: Options.level0_slowdown_writes_trigger: 20,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.level0_stop_writes_trigger: 36,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.target_file_size_base: 67108864,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.target_file_size_multiplier: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_base: 268435456,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl⁰: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl¹: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl²: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl³: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl⁴: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl⁵: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl⁶: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_sequential_skip_in_iterations: 8,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_compaction_bytes: 1677721600,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.arena_block_size: 4194304,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.disable_auto_compactions: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_style: kCompactionStyleLevel,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.size_ratio: 1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.table_properties_collectors: ,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.inplace_update_support: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.inplace_update_num_locks: 10000,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.memtable_whole_key_filtering: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.memtable_huge_page_size: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.bloom_locality: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.max_successive_merges: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.optimize_filters_for_hits: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.paranoid_file_checks: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.force_consistency_checks: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.report_bg_io_stats: 0,
debug 2019-09-05 23:09:41.840 7f58630cc200 4 rocksdb: Options.ttl: 0,
debug 2019-09-05 23:09:41.848 7f58630cc200 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.,
debug 2019-09-05 23:09:41.848 7f58630cc200 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-vdicnode05/store.db/MANIFEST-000122 succeeded,manifest_file_number is 122, next_file_number is 124, last_sequence is 232, log_number is 121,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0,
,
debug 2019-09-05 23:09:41.848 7f58630cc200 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 121,
,
debug 2019-09-05 23:09:41.848 7f58630cc200 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781849513, "job": 1, "event": "recovery_started", "log_files": [123]},
debug 2019-09-05 23:09:41.848 7f58630cc200 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #123 mode 2,
debug 2019-09-05 23:09:41.863 7f58630cc200 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781864342, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 124, "file_size": 1335, "table_properties": {"data_size": 433, "index_size": 28, "filter_size": 69, "raw_key_size": 67, "raw_average_key_size": 22, "raw_value_size": 358, "raw_average_value_size": 119, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}},
debug 2019-09-05 23:09:41.863 7f58630cc200 4 rocksdb: [db/version_set.cc:3036] Creating manifest 125,
,
debug 2019-09-05 23:09:41.863 7f58630cc200 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.,
debug 2019-09-05 23:09:41.896 7f58630cc200 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781897299, "job": 1, "event": "recovery_finished"},
debug 2019-09-05 23:09:41.932 7f58630cc200 4 rocksdb: DB pointer 0x556b88946400,
debug 2019-09-05 23:09:41.932 7f58630cc200 0 mon.vdicnode05 does not exist in monmap, will attempt to join an existing cluster,
debug 2019-09-05 23:09:41.934 7f58511a4700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 3] Compacting 4@0 + 1@6 files to L6, score 1.00,
debug 2019-09-05 23:09:41.934 7f58511a4700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 2 Base level 0, inputs: [124(1335B) 118(1335B) 112(1335B) 106(1335B)], [102(94KB)],
,
debug 2019-09-05 23:09:41.934 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781935270, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [124, 118, 112, 106], "files_L6": [102], "score": 1, "input_data_size": 101861},
debug 2019-09-05 23:09:41.935 7f58630cc200 0 using public_addr v2:192.168.100.105:0/0 -> [v2:192.168.100.105:3300/0,v1:192.168.100.105:6789/0],
debug 2019-09-05 23:09:41.938 7f5848993700 4 rocksdb: [db/db_impl.cc:777] ------ DUMPING STATS -------,
debug 2019-09-05 23:09:41.938 7f5848993700 4 rocksdb: [db/db_impl.cc:778] ,

• DB Stats **,
  Uptime(secs): 0.1 total, 0.1 interval,
  Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s,
  Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s,
  Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent,
  Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s,
  Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s,
  Interval stall: 00:00:0.000 H:M:S, 0.0 percent,
  ,
• Compaction Stats [default] **,
  Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop,
  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------,
  L0 4/4 5.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  L6 1/1 94.26 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0,
  Sum 5/5 99.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  ,
• Compaction Stats [default] **,
  Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop,
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------,
  User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  Uptime(secs): 0.1 total, 0.1 interval,
  Flush(GB): cumulative 0.000, interval 0.000,
  AddFile(GB): cumulative 0.000, interval 0.000,
  AddFile(Total Files): cumulative 0, interval 0,
  AddFile(L0 Files): cumulative 0, interval 0,
  AddFile(Keys): cumulative 0, interval 0,
  Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds,
  Interval compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds,
  Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count,
  ,
• File Read Latency Histogram By Level [default] **,
  ,
• Compaction Stats [default] **,
  Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop,
  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------,
  L0 4/4 5.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  L6 1/1 94.26 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0,
  Sum 5/5 99.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0,
  ,
• Compaction Stats [default] **,
  Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop,
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------,
  User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.01 0.00 1 0.015 0 0,
  Uptime(secs): 0.1 total, 0.0 interval,
  Flush(GB): cumulative 0.000, interval 0.000,
  AddFile(GB): cumulative 0.000, interval 0.000,
  AddFile(Total Files): cumulative 0, interval 0,
  AddFile(L0 Files): cumulative 0, interval 0,
  AddFile(Keys): cumulative 0, interval 0,
  Cumulative compaction: 0.00 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds,
  Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds,
  Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count,
  ,
• File Read Latency Histogram By Level [default] **,
  ,
  debug 2019-09-05 23:09:41.938 7f58630cc200 0 starting mon.vdicnode05 rank -1 at public addrs [v2:192.168.100.105:3300/0,v1:192.168.100.105:6789/0] at bind addrs [v2:192.168.100.105:3300/0,v1:192.168.100.105:6789/0] mon_data /var/lib/ceph/mon/ceph-vdicnode05 fsid 59457501-8a19-4a08-93f7-dd3a99db21c5,
  debug 2019-09-05 23:09:41.951 7f58511a4700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3] Generated table #129: 115 keys, 97257 bytes,
  debug 2019-09-05 23:09:41.951 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781952202, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 129, "file_size": 97257, "table_properties": {"data_size": 95647, "index_size": 472, "filter_size": 325, "raw_key_size": 2376, "raw_average_key_size": 20, "raw_value_size": 93266, "raw_average_value_size": 811, "num_data_blocks": 17, "num_entries": 115, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}},
  debug 2019-09-05 23:09:41.951 7f58511a4700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 3] Compacted 4@0 + 1@6 files to L6 => 97257 bytes,
  debug 2019-09-05 23:09:41.951 7f58511a4700 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.,
  debug 2019-09-05 23:09:41.952 7f58630cc200 1 mon.vdicnode05@-1(?) e22 preinit fsid 59457501-8a19-4a08-93f7-dd3a99db21c5,
  debug 2019-09-05 23:09:41.952 7f58630cc200 -1 mon.vdicnode05@-1(?) e22 not in monmap and have been in a quorum before; must have been removed,
  debug 2019-09-05 23:09:41.952 7f58630cc200 -1 mon.vdicnode05@-1(???) e22 commit suicide!,
  debug 2019-09-05 23:09:41.952 7f58630cc200 -1 failed to initialize,
  debug 2019-09-05 23:09:41.964 7f58511a4700 4 rocksdb: (Original Log Time 2019/09/05-23:09:41.965172) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 6.0 rd, 5.7 wr, level 6, files in(4, 1) out(1) MB in(0.0, 0.1) out(0.1), read-write-amplify(37.3) write-amplify(18.2) OK, records in: 123, records dropped: 8 output_compression: NoCompression,
  ,
  debug 2019-09-05 23:09:41.964 7f58511a4700 4 rocksdb: (Original Log Time 2019/09/05-23:09:41.965216) EVENT_LOG_v1 {"time_micros": 1567717781965199, "job": 3, "event": "compaction_finished", "compaction_time_micros": 16941, "compaction_time_cpu_micros": 2080, "output_level": 6, "num_output_files": 1, "total_output_size": 97257, "num_input_records": 123, "num_output_records": 115, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]},
  debug 2019-09-05 23:09:41.964 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781965407, "job": 3, "event": "table_file_deletion", "file_number": 124},
  debug 2019-09-05 23:09:41.964 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781965490, "job": 3, "event": "table_file_deletion", "file_number": 118},
  debug 2019-09-05 23:09:41.964 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781965592, "job": 3, "event": "table_file_deletion", "file_number": 112},
  debug 2019-09-05 23:09:41.965 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781965687, "job": 3, "event": "table_file_deletion", "file_number": 106},
  debug 2019-09-05 23:09:41.965 7f58511a4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1567717781965774, "job": 3, "event": "table_file_deletion", "file_number": 102},
  teardown: managing teardown after SIGCHLD,
  teardown: Waiting PID 631 to terminate ,
  teardown: Process 631 is terminated,
  teardown: Bye Bye, container will die with return code -1,
  teardown: if you don't want me to die and have access to a shell to debug this situation, next time run me with '-e DEBUG=stayalive',

Looking at the documentation looks /etc/ceph must not be mounted when using kv store

docker run -d --net=host \
-v /var/lib/ceph:/var/lib/ceph \
-e MON_IP=192.168.0.20 \
-e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \
-e KV_TYPE=etcd \
-e KV_IP=192.168.0.20 \
ceph/daemon mon

But looking at ths source code, maybe it must because it stores monmap in /etc/ceph

Any help will be welcome