Ceph QA

Several branches pushed couple of hours after my branch already have binaries being built for them, current build is stuck - https://shaman.ceph.com/builds/ceph/wip-rishabh-testing-20240404.111254-quincy/3b945a878566727470714de2e4e8c964e4810a88/

Builds are really slow. I've pushed 4-5 times so far and build were never triggered. Even the ones that are on are very slow - https://shaman.ceph.com/builds/ceph/wip-rishabh-testing-20240404.111254-quincy/2a95c7615793a5b36f31e2064df782c93dc2e79e/

Despite of the 3 pushes and despite of many build jobs being gren, packages are not available. Got more than 100 messages like these -

2024-04-05 09:23:19,225.225 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,443.443 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,456.456 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,466.466 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,478.478 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,487.487 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,496.496 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found
2024-04-05 09:23:19,505.505 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0971ad0070783d7c961c93dd9dbc6e413d7576aa' not found

2024-04-05 09:23:27,920.920 INFO:teuthology.suite.util:Missing Packages: /home/rishabh/repos/teuthology/venv-teuth/bin/teuthology-schedule --name rishabh-2024-04-05_09:23:02-fs-wip-rishabh-testing-20240404.111254-quincy-testing-default-smithi --worker smithi --dry-run --priority 70 --num 1 --description 'fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}' -- /tmp/schedule_suite_fe1hw4l_ -
2024-04-05 09:23:27,921.921 INFO:teuthology.suite.run:Suite fs in /cephfs/home/rishabh/src/git.ceph.com_ceph-c_wip-rishabh-testing-20240404.111254-quincy/qa/suites/fs scheduled 192 jobs.
2024-04-05 09:23:27,921.921 INFO:teuthology.suite.run:12/204 jobs were filtered out.
2024-04-05 09:23:27,921.921 WARNING:teuthology.suite.run:Scheduled 192/192 jobs that are missing packages!
2024-04-05 09:23:27,921.921 INFO:teuthology.suite.run:Scheduled 192 jobs in total.

https://pulpito.ceph.com/rishabh-2024-04-11_13:44:02-fs-wip-rishabh-testing-20240404.111254-quincy-testing-default-smithi/7652302
https://pulpito.ceph.com/rishabh-2024-04-11_16:20:16-fs-wip-rishabh-testing-20240404.111254-quincy-testing-default-smithi/7652324
https://pulpito.ceph.com/rishabh-2024-04-11_17:12:03-fs-wip-rishabh-testing-20240404.111254-quincy-testing-default-smithi/7652521
test_multiple_path_r was fixed here: https://pulpito.ceph.com/rishabh-2024-04-11_18:02:31-fs-wip-rishabh-testing-20240404.111254-quincy-testing-default-smithi/7652537