Ceph QA

So far, the issue is not obvious, but I'm continuing to look into it.

Going to raise a PR to clean up some of the MON_DOWN warnings, as it makes it very difficult to accurately review the suite.

Reran tests here to try and clear the mon warnings: https://pulpito.ceph.com/lflores-2024-04-01_18:07:25-rados-wip-yuri8-testing-2024-03-25-1419-distro-default-smithi/

(Tested with https://github.com/ceph/ceph/pull/56619 in the qa suite)

lflores pls take a look (using the updated Tracker functionality)

@Yuri Weinstein checking on this

Failure, unrelated :
1. cephadm: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c in cluster log
2. cephadm: Health check failed: 1 osds down (OSD_DOWN) in cluster log
3. qa/standalone/scrub/osd-scrub-repair.sh: TEST_repair_stats_ec: test 26 = 13
4. cephadm: log_channel(cephadm) log [ERR] : Failed to connect to smithi090 (10.0.0.9). Permission denied
5. rados/thrash: AssertionError: wait_for_recovery: failed before timeout expired
6. cephadm/osds, cephadm/workunits: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) in cluster log
7. ceph_test_lazy_omap_stats segfault while waiting for active+clean
8. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons
9. rados/cephadm/smoke: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) in cluster log
10. Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement)
- Orchestrator
11. Add expected warnings in cluster log to ignorelists - RADOS
12. mon/health-mute.sh: TEST_mute: return 1 (HEALTH WARN 3 mgr modules have failed dependencies) - RADOS
13. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - RADOS

Details:
1. https://tracker.ceph.com/issues/64864 <------- New Tracker
2. https://tracker.ceph.com/issues/64865
3. https://tracker.ceph.com/issues/64437
4. https://tracker.ceph.com/issues/65017
5. https://tracker.ceph.com/issues/63198
6. https://tracker.ceph.com/issues/64868
7. https://tracker.ceph.com/issues/59196
8. https://tracker.ceph.com/issues/61774
9. https://tracker.ceph.com/issues/64872
10. https://tracker.ceph.com/issues/64374
11. https://tracker.ceph.com/issues/65521
12. https://tracker.ceph.com/issues/54515
13. https://tracker.ceph.com/issues/59380

@Prashant D if you approved this batch pls change the status and assign to me for merge