Feature #63801
openverified mon backups
0%
Description
Currently there is no nice solution for getting consistent, verified mon backups.
Most simply stop one mon, copy the var folder and restart the mon.
I suggest following enhancments:
Add backup directory in mon settings in which backups will be stored.
Add a mon admin command to trigger a backup in the configured directory and
return the name of the backup.
When triggered, it uses the rocksdb BackupEngine to create a verified backup.
I think having verifed backup of mon is very important for restoration of a damaged cluster.
If OSD encryption is used, recovery of mon from OSDs is impossible.
Updated by Daniel Poelzleithner 5 months ago
I'm planning on implementing this, I just wanted to get feedback before implementing.
Updated by Christian Rohmann 25 days ago
This is really a good idea to have built-in! Thanks for taking this up!
We have been using a custom backup script utilizing ceph config-key dump to at least dump the OSD encryption keys to have them in case the mons are destroyed or their database becomes corrupted.
- Full restore might not always be wanted, so extraction of e.g. "only OSD encryption keys" should be possible / documented maybe?
- There should be an automatic mechanism to create and rotate backups (very basic logic ... interval + number of copies to keep), this removed all the need for additional tooling (cron, systemd-timers, rotation, ...)
Updated by Daniel Poelzleithner 24 days ago
Christian Rohmann wrote in #note-2:
My thoughts would be:
- Full restore might not always be wanted, so extraction of e.g. "only OSD encryption keys" should be possible / documented maybe?
I will investigate this later. I have to look how the encryption keys are actually stored. Since I use the rockdb backup functions, they are more like everything or nothing.
I will look if there is a way to open the backup and extract keys by hand, if so, I will implement a key merge.
- There should be an automatic mechanism to create and rotate backups (very basic logic ... interval + number of copies to keep), this removed all the need for additional tooling (cron, systemd-timers, rotation, ...)
I'm working on the cleanup routine right now, since I want some proper time windows in which backups exist. Like 1 backup 7 days old, one for every day and the last 5 backups.