Ceph

The manual installation instructions for Adding/Removing Monitors (https://docs.ceph.com/en/reef/rados/operations/add-or-rm-mons/) says you should run this when adding a second/third monitor:

$ sudo ceph-mon -i {mon-id} --mkfs --monmap {tmp}/{map-filename} --keyring {tmp}/{key-filename}

which will make the store.db and other files root-owned. This in turn crashes the mon at start with this output:

root@REDACTED-mon-2:/var/lib/ceph/mon# /usr/bin/ceph-mon -f --cluster ceph --id REDACTED-mon-2 --setuser ceph --setgroup ceph
2023-08-17T13:51:50.341+0000 7f794ec73a00 -1 open error initializing leveldb db back storage in /var/lib/ceph/mon/ceph-REDACTED-mon-2/store.db
/build/ceph-17.2.6/src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7f794ec73a00 time 2023-08-17T13:51:50.346917+0000
/build/ceph-17.2.6/src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: (ceph::_ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe0) [0x7f794fb43fc3]
2: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x88f) [0x55c5187d9c5f]
3: (MonitorDBStore::open(std::ostream&)+0x170) [0x55c5187da200]
4: main()
5: __libc_start_main()
6: _start()

• Caught signal (Aborted) **
  in thread 7f794ec73a00 thread_name:ceph-mon
  2023-08-17T13:51:50.345+0000 7f794ec73a00 -1 /build/ceph-17.2.6/src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7f794ec73a00 time 2023-08-17T13:51:50.346917+0000
  /build/ceph-17.2.6/src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")

ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
(lots more...)

So, docs are wrong + untested but also core-dumping on a simple permission-denied in the filesystem is kind of bad feedback.

chown:ing the files to ceph:ceph after the mon --mkfs run fixes it so it works again.