Ceph

Sometimes, users would like to prioritise some pools over others during recovery.

For example, in Ceph FS it is more important to recover the metadata pool's PGs (because until that happens it is likely that no filesystem work can continue) than to recover the data pool's PGs (because missing data objects usually only hurt any clients accessing those particular files/objects).

We can also imagine that a customer might consider their RBD pool for VM images as higher priority than an RGW pools used for application data.

This would be a user-configurable thing, but we would probably sometimes set relative priorities by default during e.g. "fs new". It should be accessible in the same get/set commands as other pool properties like pg_num. The default pool priority should be something 'middle' so that we can easily prioritise or deprioritise pools relative to the default 'normal' priority.