Ceph » Orchestrator

- ceph orch upgrade should set noout, nodeep-scrub, and noscrub and unset when the upgrade will complete
- This was the case when we used to use ceph-ansible
- This is a kind of feature parity b/w ceph-ansible and cephadm

- This feature can be designed as optional with default as True so if some users/admins do not want then can set it to false.

Benefits:

1. Less load from scrubbing during the upgrade when we expect to have recovery in the cluster
2. If an OSD is taking longer to reboot -> boot due to different issues¹ or slow boot

[1] For example, PG dups issue - https://tracker.ceph.com/issues/53729 - it takes approx 7 to 8 minutes for an NVMe OSD to boot with 50M dups and approx 12-15 minutes for hybrid HDD OSDs
and if an OSD takes more than 10 minutes the Monitor marks the down OSD out and we will have backfill/recovery in the cluster when the upgrade is running and we do not want that
There can be multiple examples hence running the upgrade with the following flags is recommended:

noscrub
nodeep-scrub
noout