Ceph » RADOS

h1. MAIN

Summaries are ordered latest --> oldest.

h3. https://trello.com/c/k9RNURve/1897-wip-yuri3-testing-2023-12-07-0727-old-wip-yuri3-testing-wip-neorados-learning-from-experience

https://pulpito.ceph.com/cbodley-2023-12-15_15:26:38-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/

7493218  - https://tracker.ceph.com/issues/63783 (known issue)

7493242
7493228

——————


7493219 - https://tracker.ceph.com/issues/6177 (known issue)

7493224 

7493230

7493236

7493237

7493243


———————


7493221 - https://tracker.ceph.com/issues/59142 (known issue)


———————

7493223 - https://tracker.ceph.com/issues/59196 (known issue)

———————


7493229 -  https://tracker.ceph.com/issues/63748 (known issue)

7493244

———————

7493232 - https://tracker.ceph.com/issues/63785 (known issue)

h3. https://trello.com/c/ZgZGBobg/1902-wip-yuri8-testing-2023-12-11-1101-old-wip-yuri8-testing-2023-12-06-1425

https://pulpito.ceph.com/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/

Unrelated failures:

1. https://tracker.ceph.com/issues/63748

2. https://tracker.ceph.com/issues/61774

3. https://tracker.ceph.com/issues/63783

4. https://tracker.ceph.com/issues/59380

5. https://tracker.ceph.com/issues/59142

6. https://tracker.ceph.com/issues/59196

Details:

1. ['7487680', '7487836'] - qa/workunits/post-file.sh: Couldn't create directory

2. ['7487541', '7487610', '7487750', '7487751', '7487683', '7487822'] - centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons

3. ['7487677', '7487816', '7487532'] - mgr: 'ceph rgw realm bootstrap' fails with KeyError: 'realm_name'

4. ['7487804', '7487806', '7487647'] - rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

5. ['7487724', '7487568'] - mgr/dashboard: fix e2e for dashboard v3

6. ['7487579', '7487739'] - cephtest bash -c ceph_test_lazy_omap_stats

h3. https://trello.com/c/OeXUIG19/1898-wip-yuri2-testing-2023-12-06-1239-old-wip-yuri2-testing-2023-12-04-0902

https://pulpito.ceph.com/?branch=wip-yuri2-testing-2023-12-06-1239

Failures, unrelated:

    1. https://tracker.ceph.com/issues/63748

    2. https://tracker.ceph.com/issues/56788

    3. https://tracker.ceph.com/issues/61774

    4. https://tracker.ceph.com/issues/63783

    5. https://tracker.ceph.com/issues/63784 -- new tracker

    6. https://tracker.ceph.com/issues/59196

    7. https://tracker.ceph.com/issues/63785 -- new tracker

    8. https://tracker.ceph.com/issues/59380

    9. https://tracker.ceph.com/issues/63788

    10. https://tracker.ceph.com/issues/63778

    11. https://tracker.ceph.com/issues/63789

Details:

    1. qa/workunits/post-file.sh: Couldn't create directory - Infrastructure

    2. crash: void KernelDevice::_aio_thread(): abort - Ceph - Bluestore

    3. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    4. mgr: 'ceph rgw realm bootstrap' fails with KeyError: 'realm_name' - Ceph - RGW

    5. qa/standalone/mon/mkfs.sh:'mkfs/a' already exists and is not empty: monitor may already exist - Ceph - RADOS

    6. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    7. cephadm/test_adoption.sh: service not found - Ceph - Orchestrator

    8. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    9. Cephadm tests fail from "nothing provides lua-devel needed by ceph-2:19.0.0-44.g2d90d175.el8.x86_64" - Ceph - RGW

    10. Upgrade: failed due to an unexpected exception - Ceph - Orchestrator

    11. LibRadosIoEC test failure - Ceph - RADOS

h3. https://trello.com/c/wK3QrkV2/1901-wip-yuri-testing-2023-12-06-1240

https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-12-06-1240

Failures, unrelated:

    1. https://tracker.ceph.com/issues/63783

    2. https://tracker.ceph.com/issues/61774

    3. https://tracker.ceph.com/issues/59196

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/59142

    6. https://tracker.ceph.com/issues/63748

    7. https://tracker.ceph.com/issues/63785 -- new tracker

    8. https://tracker.ceph.com/issues/63786 -- new tracker

Details:

    1. mgr: 'ceph rgw realm bootstrap' fails with KeyError: 'realm_name' - Ceph - RGW

    2. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    3. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    5. mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

    6. qa/workunits/post-file.sh: Couldn't create directory - Infrastructure

    7. cephadm/test_adoption.sh: service not found - Ceph - Orchestrator

    8. rados_cls_all: TestCls2PCQueue.MultiProducer hangs - Ceph - RGW

h3. https://trello.com/c/tUEWtLfq/1892-wip-yuri7-testing-2023-11-17-0819

https://pulpito.ceph.com/yuriw-2023-11-26_21:30:23-rados-wip-yuri7-testing-2023-11-17-0819-distro-default-smithi/

Failures being analyzed:

1. '7467376' - ?

2. 

Failures, unrelated:

1. ['7467380','7467367','7467378'] - timeout on test_cls_2pc_queue ->> https://tracker.ceph.com/issues/62449

2. ['7467370','7467374','7467387','7467375'] - Valgrind: mon (Leak_StillReachable) ->> https://tracker.ceph.com/issues/61774

3. ['7467388','7467373'] - ceph_test_lazy_omap_stats ->> https://tracker.ceph.com/issues/59196

4. ['7467385','7467372'] - failure in e2e-spec ->> https://tracker.ceph.com/issues/48406

5. ['7467371'] - unrelated. Test infra issues.

6. ['7467379','7467369'] - RGW 'realm_name' ->> https://tracker.ceph.com/issues/63499

7. ['7467377','7467366'] - seem to be a disk space issue.

8. ['7467381'] - test environment

h3. https://pulpito.ceph.com/yuriw-2023-10-24_00:11:03-rados-wip-yuri2-testing-2023-10-23-0917-distro-default-smithi/

Failures, unrelated:

1. https://tracker.ceph.com/issues/59196

2. https://tracker.ceph.com/issues/61774

3. https://tracker.ceph.com/issues/49961

4. https://tracker.ceph.com/issues/62449

5. https://tracker.ceph.com/issues/48406

6. https://tracker.ceph.com/issues/63121

7. https://tracker.ceph.com/issues/47838

8. https://tracker.ceph.com/issues/62777

9. https://tracker.ceph.com/issues/54372

10. https://tracker.ceph.com/issues/63500      <--------- New tracker

Details:

1. ['7435483','7435733'] - ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

2. ['7435516','7435570','7435765','7435905'] - centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

3. ['7435520'] - scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed - Ceph - RADOS

4. ['7435568','7435875'] - test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure

5. ['7435713','7436026'] - cephadm/test_dashboard_e2e.sh: error when testing orchestrator/04-osds.e2e-spec.ts - Ceph - Mgr - Dashboard

6. ['7435741'] - objectstore/KeyValueDB/KVTest.RocksDB_estimate_size tests failing - Ceph - RADOS

7. ['7435855'] - mon/test_mon_osdmap_prune.sh: first_pinned != trim_to - Ceph - RADOS

8. ['7435767', '7435636','7435971'] - rados/valgrind-leaks: expected valgrind issues and found none - Ceph - RADOS

9. ['7435999'] - No module named 'tasks' - Infrastructure

10. ['7435995'] - No module named 'tasks.nvme_loop' - Infrastructure

h3. https://pulpito.ceph.com/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/

Failures, unrelated:

1. https://tracker.ceph.com/issues/59380 

2. https://tracker.ceph.com/issues/59142

3. https://tracker.ceph.com/issues/61774

4. https://tracker.ceph.com/issues/53767

5. https://tracker.ceph.com/issues/59196

6. https://tracker.ceph.com/issues/62535

Details:

1. ['7441165', '7441319'] - rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

2. ['7441240', '7441396'] - mgr/dashboard: fix e2e for dashboard v3

3. ['7441266', '7441336', '7441129', '7441267', '7441201'] - centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

4. ['7441167', '7441321'] - qa/workunits/cls/test_cls_2pc_queue.sh: killing an osd during thrashing causes timeout - Ceph - RADOS

5. ['7441250', '7441096'] - ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

6. ['7441374'] - cephadm: wait for healthy state times out because cephadm agent is down - Ceph - Orchestrator

h3. https://pulpito.ceph.com/yuriw-2023-10-27_19:03:28-rados-wip-yuri8-testing-2023-10-27-0825-distro-default-smithi/

Failures, unrelated:

1. https://tracker.ceph.com/issues/59196

2. https://tracker.ceph.com/issues/61774

3. https://tracker.ceph.com/issues/62449

4. https://tracker.ceph.com/issues/59192

5. https://tracker.ceph.com/issues/48406

6. https://tracker.ceph.com/issues/62776

Details:

1. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

2. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

3. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure

4. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

5. cephadm/test_dashboard_e2e.sh: error when testing orchestrator/04-osds.e2e-spec.ts - Ceph - Mgr - Dashboard

6. rados/basic: 2 pools do not have an application enabled - Ceph - RADOS

h3. Not in Trello but still a rados suite

https://pulpito.ceph.com/ksirivad-2023-10-13_01:58:36-rados-wip-ksirivad-fix-63183-distro-default-smithi/

Failures, unrelated:

7423809 - https://tracker.ceph.com/issues/63198 <<-- New Tracker

7423821, 7423972 - https://tracker.ceph.com/issues/59142 - mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

7423826, 7423979 - https://tracker.ceph.com/issues/59196 - ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

7423849 - https://tracker.ceph.com/issues/61774 - centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

7423875 - Failed to get package from Shaman (infra failure)

7423896, 7424047 - https://tracker.ceph.com/issues/62449 - test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure

7423913 - https://tracker.ceph.com/issues/62119 - timeout on reserving replica

7423918 - https://tracker.ceph.com/issues/61787 - Command "ceph --cluster ceph osd dump --format=json" times out when killing OSD

7423980 - https://tracker.ceph.com/issues/62557 - Teuthology test failure due to "MDS_CLIENTS_LAGGY" warning

7423982 - https://tracker.ceph.com/issues/63121 - KeyValueDB/KVTest.RocksDB_estimate_size tests failing

7423984 - tracker.ceph.com/issues/61774 - centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons

7423995 - https://tracker.ceph.com/issues/62777 - rados/valgrind-leaks: expected valgrind issues and found none

7424052 - https://tracker.ceph.com/issues/55809 - "Leak_IndirectlyLost" valgrind report on mon.c

h3. https://trello.com/c/PuCOnhYL/1841-wip-yuri5-testing-2023-10-02-1105-old-wip-yuri5-testing-2023-09-27-0959

https://pulpito.ceph.com/?branch=wip-yuri5-testing-2023-10-02-1105

Failures, unrelated:

    1. https://tracker.ceph.com/issues/52624

    2. https://tracker.ceph.com/issues/59380

    3. https://tracker.ceph.com/issues/61774

    4. https://tracker.ceph.com/issues/59196

    5. https://tracker.ceph.com/issues/62449

    6. https://tracker.ceph.com/issues/59142

Details:

    1. qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - Ceph - RADOS

    2. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    3. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    4. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    5. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

    6. mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

h3. https://trello.com/c/tGZO7hNK/1832-wip-yuri6-testing-2023-08-28-1308

https://pulpito.ceph.com/lflores-2023-09-06_18:19:12-rados-wip-yuri6-testing-2023-08-28-1308-distro-default-smithi/

Failures, unrelated:

# https://tracker.ceph.com/issues/59142

# https://tracker.ceph.com/issues/59196

# https://tracker.ceph.com/issues/55347

# https://tracker.ceph.com/issues/62084

# https://tracker.ceph.com/issues/62975 <<<---- New Tracker

# https://tracker.ceph.com/issues/62449

# https://tracker.ceph.com/issues/61774

# https://tracker.ceph.com/issues/53345

Details:

# mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

# ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

# SELinux Denials during cephadm/workunits/test_cephadm

# task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

# site-packages/paramiko/channel.py: OSError: Socket is closed - Infrastructure

# test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

# centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

# Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) - Ceph - Orchestrator

h3. Individual testing for https://github.com/ceph/ceph/pull/53344

http://pulpito.front.sepia.ceph.com/?branch=wip-lflores-testing-2-2023-09-08-1755

Failures, unrelated:

    1. https://tracker.ceph.com/issues/57628

    2. https://tracker.ceph.com/issues/62449

    3. https://tracker.ceph.com/issues/61774

    4. https://tracker.ceph.com/issues/53345

    5. https://tracker.ceph.com/issues/59142

    6. https://tracker.ceph.com/issues/59380

Details:

    1. osd:PeeringState.cc: FAILED ceph_assert(info.history.same_interval_since != 0) - Ceph - RADOS

    2. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

    3. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    4. Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) - Ceph - Orchestrator

    5. mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

    6. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

h3. https://trello.com/c/JxeRJYse/1822-wip-yuri4-testing-2023-08-10-1739

https://pulpito.ceph.com/?branch=wip-yuri4-testing-2023-08-10-1739

Failures, unrelated:

    1. https://tracker.ceph.com/issues/62084

    2. https://tracker.ceph.com/issues/59192

    3. https://tracker.ceph.com/issues/61774

    4. https://tracker.ceph.com/issues/62449

    5. https://tracker.ceph.com/issues/62777

    6. https://tracker.ceph.com/issues/59196

    7. https://tracker.ceph.com/issues/59380

    8. https://tracker.ceph.com/issues/58946

Details:

    1. task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - CephFS

    2. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    3. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    4. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

    5. rados/valgrind-leaks: expected valgrind issues and found none - Ceph - RADOS

    6. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    7. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    8. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

h3. https://trello.com/c/Wt1KTViI/1830-wip-yuri-testing-2023-08-25-0809

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri-testing-2023-08-25-0809

Failures, unrelated:

    1. https://tracker.ceph.com/issues/62776

    2. https://tracker.ceph.com/issues/61774

    3. https://tracker.ceph.com/issues/62084

    4. https://tracker.ceph.com/issues/58946

    5. https://tracker.ceph.com/issues/59380

    6. https://tracker.ceph.com/issues/62449

    7. https://tracker.ceph.com/issues/59196

Details:

    1. rados/basic: 2 pools do not have an application enabled - Ceph - RADOS

    2. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    3. task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

    4. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    5. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    6. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

    7. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

h3. https://trello.com/c/Fllj7bVM/1833-wip-yuri8-testing-2023-08-28-1340

https://pulpito.ceph.com/?branch=wip-yuri8-testing-2023-08-28-1340

Failures, unrelated:

    1. https://tracker.ceph.com/issues/62728

    2. https://tracker.ceph.com/issues/62084

    3. https://tracker.ceph.com/issues/59142

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/62449

    6. https://tracker.ceph.com/issues/61774

    7. https://tracker.ceph.com/issues/59196

Details:

    1. Host key for server xxx does not match - Infrastructure

    2. task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

    3. mgr/dashboard: fix e2e for dashboard v3 - Ceph - Mgr - Dashboard

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    5. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

    6. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    7. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

h3. https://trello.com/c/pMEWaauy/1825-wip-yuri3-testing-2023-08-15-0955

Failures, unrelated:

7370238 - https://tracker.ceph.com/issues/59196

7370242, 7370322 - https://tracker.ceph.com/issues/62482

7370245,7370298 - https://tracker.ceph.com/issues/62084

7370250, 7370317 - https://tracker.ceph.com/issues/53767

7370274, 7370343 - https://tracker.ceph.com/issues/61519

7370263 - https://tracker.ceph.com/issues/62713 (New tracker)

7370285, 7370286 - https://tracker.ceph.com/issues/61774

DEAD jobs, unrelated:

7370249, 7370316 https://tracker.ceph.com/issues/59380

h3. https://trello.com/c/i87i4GUf/1826-wip-yuri10-testing-2023-08-17-1444-old-wip-yuri10-testing-2023-08-15-1601-old-wip-yuri10-testing-2023-08-15-1009

Failures, unrelated:

7376678, 7376832 - https://tracker.ceph.com/issues/61786

7376687 - https://tracker.ceph.com/issues/59196

7376699 - https://tracker.ceph.com/issues/55347

7376739 - https://tracker.ceph.com/issues/61229

7376742, 7376887 - https://tracker.ceph.com/issues/62084

7376758, 7376914 - https://tracker.ceph.com/issues/62449

7376756, 7376912 - https://tracker.ceph.com/issues/59380

https://tracker.ceph.com/issues/61774

h3. https://trello.com/c/MEs20HAJ/1828-wip-yuri11-testing-2023-08-17-0823

https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-08-17-0823

Failures, unrelated:

    1. https://tracker.ceph.com/issues/47838

    2. https://tracker.ceph.com/issues/61774

    3. https://tracker.ceph.com/issues/59196

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/62535 -- new tracker

    7. https://tracker.ceph.com/issues/62084

    8. https://tracker.ceph.com/issues/62449

Details:

    1. mon/test_mon_osdmap_prune.sh: first_pinned != trim_to - Ceph - RADOS

    2. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    3. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    6. cephadm: wait for healthy state times out because cephadm agent is down - Ceph - Orchestrator

    7. task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

    8. test/cls_2pc_queue: TestCls2PCQueue.MultiProducer and TestCls2PCQueue.AsyncConsumer failure - Ceph - RGW

h3. https://pulpito.ceph.com/?branch=wip-yuri7-testing-2023-07-20-0727

Failures, unrelated:

    1. https://tracker.ceph.com/issues/62084

    2. https://tracker.ceph.com/issues/61161

    3. https://tracker.ceph.com/issues/61774

    4. https://tracker.ceph.com/issues/62167

    5. https://tracker.ceph.com/issues/62212

    6. https://tracker.ceph.com/issues/58946

    7. https://tracker.ceph.com/issues/59196

    8. https://tracker.ceph.com/issues/59380

Details:

    1. task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

    2. Creating volume group 'vg_nvme' failed - Ceph - Ceph-Ansible

    3. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    4. FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) - Ceph - RADOS

    5. cannot create directory ‘/home/ubuntu/cephtest/archive/audit’: No such file or directory - Tools - Teuthology

    6. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    7. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    8. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

h3. WIP https://trello.com/c/1JlLNnGN/1812-wip-yuri5-testing-2023-07-24-0814

https://pulpito.ceph.com/?branch=wip-yuri5-testing-2023-07-24-0814

Failures:

    1. 7350969, 7351069 -> https://tracker.ceph.com/issues/59192

    2. 7350972, 7351023 -> https://tracker.ceph.com/issues/62073  

    3. 7350977, 7351059 -> https://tracker.ceph.com/issues/53767 ? to verify

    4. 7350983, 7351016, 7351019 -> https://tracker.ceph.com/issues/61774 (valgrind)

    5. 7351004, 7351090 -> https://tracker.ceph.com/issues/61519

    6. 7351084 -> (selinux)

Dead:

    7. 7350974, 7351053 -> no relevant info.

Details:

    1.  cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    2.  AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

    3.  e2e - to verify

    4.  valgrind issues. Won't be analyzed as irrelevant to this PR.

    5.  mgr/dashboard: fix test_dashboard_e2e.sh failure - Ceph - Mgr - Dashboard

    6.  selinux issue: 

h3. https://trello.com/c/GM3omhGs/1803-wip-yuri5-testing-2023-07-14-0757-old-wip-yuri5-testing-2023-07-12-1122

https://pulpito.ceph.com/?branch=wip-yuri5-testing-2023-07-14-0757

Failures:

    1. 7341711 -> https://tracker.ceph.com/issues/62073 -> AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd'

    2. 7341716 -> https://tracker.ceph.com/issues/58946 -> cephadm: KeyError: 'osdspec_affinity'

    3. 7341717 -> https://tracker.ceph.com/issues/62073 -> AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd'

    4. 7341720 -> https://tracker.ceph.com/issues/58946 -> cephadm: KeyError: 'osdspec_affinity'

Dead:

    1. 7341712 -> https://tracker.ceph.com/issues/59380 -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

    2. 7341719 -> https://tracker.ceph.com/issues/59380 -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

h3. https://trello.com/c/6no1SSqS/1762-wip-yuri2-testing-2023-07-17-0957-old-wip-yuri2-testing-2023-07-15-0802-old-wip-yuri2-testing-2023-07-13-1236-old-wip-yuri2-test

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri2-testing-2023-07-15-0802

Failures:

    1. https://tracker.ceph.com/issues/61774 -- valgrind leak in centos 9; not major outside of qa but needs to be suppressed

    2. https://tracker.ceph.com/issues/58946

    3. https://tracker.ceph.com/issues/59192

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/61385

    6. https://tracker.ceph.com/issues/62073

Details:

    1. centos 9 testing reveals rocksdb "Leak_StillReachable" memory leak in mons - Ceph - RADOS

    2. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    3. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    5. TEST_dump_scrub_schedule fails from "key is query_active: negation:0 # expected: true # in actual: false" - Ceph - RADOS

    6. AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

h3. https://trello.com/c/Vbn2qwgo/1793-wip-yuri-testing-2023-07-14-1641-old-wip-yuri-testing-2023-07-12-1332-old-wip-yuri-testing-2023-07-12-1140-old-wip-yuri-testing

https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-07-14-1641

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/59380

    3. https://tracker.ceph.com/issues/62073

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    2. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    3. AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' - Ceph - CephFS

h3. https://trello.com/c/7y6uj4bo/1800-wip-yuri6-testing-2023-07-10-0816

Failures:

7332480 -> https://tracker.ceph.com/issues/58946 -> cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

7332558 -> https://tracker.ceph.com/issues/57302 -> Test failure: test_create_access_permissions (tasks.mgr.dashboard.test_pool.PoolTest)

7332565 -> https://tracker.ceph.com/issues/57754 -> test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure

7332612 -> https://tracker.ceph.com/issues/57754 -> test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure

7332613 -> https://tracker.ceph.com/issues/55347 -> SELinux Denials during cephadm/workunits/test_cephadm

7332636 -> https://tracker.ceph.com/issues/58946 -> cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

Deads:

7332357 -> https://tracker.ceph.com/issues/61164 -> Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds

7332405 -> https://tracker.ceph.com/issues/59380 -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

7332559 -> https://tracker.ceph.com/issues/59380 -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

h3. https://trello.com/c/BHAY6fGO/1801-wip-yuri10-testing-2023-07-10-1345

https://pulpito.ceph.com/?branch=wip-yuri10-testing-2023-07-10-1345

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/50242

    3. https://tracker.ceph.com/issues/55347

    4. https://tracker.ceph.com/issues/59380

Details :

    1 cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    2. test_repair_corrupted_obj fails with assert not inconsistent

    3. SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

h3. https://trello.com/c/bn3IMWEB/1783-wip-yuri7-testing-2023-06-23-1022-old-wip-yuri7-testing-2023-06-12-1220-old-wip-yuri7-testing-2023-06-09-1607

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri7-testing-2023-06-23-1022

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59380

    2. https://tracker.ceph.com/issues/57754

    3. https://tracker.ceph.com/issues/59196

    4. https://tracker.ceph.com/issues/59057

    5. https://tracker.ceph.com/issues/57754

    6. https://tracker.ceph.com/issues/55347

    7. https://tracker.ceph.com/issues/58946

    8. https://tracker.ceph.com/issues/61951 -- new tracker

Details:

    1. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    2. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure

    3. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    4. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' - Ceph - RADOS

    5. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure

    6. SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure

    7. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    8. cephadm: OrchestratorError: Must set public_network config option or specify a CIDR network, ceph addrvec, or plain IP - Ceph - Orchestrator

h3. https://trello.com/c/BVxlgRvT/1782-wip-yuri5-testing-2023-06-28-1515-old-wip-yuri5-testing-2023-06-21-0750-old-wip-yuri5-testing-2023-06-16-1012-old-wip-yuri5-test

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri5-testing-2023-06-28-1515

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/59380

    3. https://tracker.ceph.com/issues/55347

    4. https://tracker.ceph.com/issues/57754

    5. https://tracker.ceph.com/issues/59057

    6. https://tracker.ceph.com/issues/61897

    7. https://tracker.ceph.com/issues/61940 -- new tracker

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    2. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    3. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    4. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Ceph - RADOS

    5. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' - Ceph - RADOS

    6. qa: rados:mgr fails with MDS_CLIENTS_LAGGY - Ceph - CephFS

    7. "test_cephfs_mirror" fails from stray cephadm daemon - Ceph - Orchestrator

h3. 2023 Jun 23

https://pulpito.ceph.com/rishabh-2023-06-21_22:15:54-rados-wip-rishabh-improvements-authmon-distro-default-smithi/

https://pulpito.ceph.com/rishabh-2023-06-22_10:54:41-rados-wip-rishabh-improvements-authmon-distro-default-smithi/

* https://tracker.ceph.com/issues/58946

  cephadm: KeyError: 'osdspec_affinity'

* https://tracker.ceph.com/issues/57754

  test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist

* https://tracker.ceph.com/issues/61784

  test_envlibrados_for_rocksdb.sh: '~ubuntu-toolchain-r' user or team does not exist

* https://tracker.ceph.com/issues/61832

  osd-scrub-dump.sh: ERROR: Extra scrubs after test completion...not expected

h3. https://trello.com/c/CcKXkHLe/1789-wip-yuri3-testing-2023-06-19-1518

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri3-testing-2023-06-19-1518

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59057

    2. https://tracker.ceph.com/issues/59380

    3. https://tracker.ceph.com/issues/58946

Details:

    1. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' - Ceph - RADOS

    2. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    3. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

h3. https://trello.com/c/Zbp7w1yE/1770-wip-yuri10-testing-2023-06-02-1406-old-wip-yuri10-testing-2023-05-30-1244

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri10-testing-2023-06-02-1406

Failures, unrelated:

    1. https://tracker.ceph.com/issues/46877

    2. https://tracker.ceph.com/issues/59057

    3. https://tracker.ceph.com/issues/61225

    4. https://tracker.ceph.com/issues/55347

    5. https://tracker.ceph.com/issues/59380

Details:

    1. mon_clock_skew_check: expected MON_CLOCK_SKEW but got none - Ceph - RADOS

    2. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    3. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    4. SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure

    5. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

h3. https://trello.com/c/lyHYQLgL/1771-wip-yuri11-testing-2023-05-30-1325

https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-05-30-1325

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59678

    2. https://tracker.ceph.com/issues/55347

    3. https://tracker.ceph.com/issues/59380

    4. https://tracker.ceph.com/issues/61519

    5. https://tracker.ceph.com/issues/61225

Details:

    1. rados/test_envlibrados_for_rocksdb.sh: Error: Unable to find a match: snappy-devel - Infrastructure

    2. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    3. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    4. mgr/dashboard: fix test_dashboard_e2e.sh failure - Ceph - Mgr - Dashboard

    5. TestClsRbd.mirror_snapshot failure - Ceph - RBD

h3. https://trello.com/c/8FwhCHxc/1774-wip-yuri-testing-2023-06-01-0746

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri-testing-2023-06-01-0746

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59380

    2. https://tracker.ceph.com/issues/61578 -- new tracker

    3. https://tracker.ceph.com/issues/59192

    4. https://tracker.ceph.com/issues/61225

    5. https://tracker.ceph.com/issues/59057

Details:

    1. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    2. test_dashboard_e2e.sh: Can't run because no spec files were found - Ceph - Mgr - Dashboard

    3. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    4. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    5. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

h3. https://trello.com/c/g4OvqEZx/1766-wip-yuri-testing-2023-05-26-1204

https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-05-26-1204

Failures, unrelated:

1. https://tracker.ceph.com/issues/61386

2. https://tracker.ceph.com/issues/61497 -- new tracker

3. https://tracker.ceph.com/issues/61225

4. https://tracker.ceph.com/issues/58560

5. https://tracker.ceph.com/issues/59057

6. https://tracker.ceph.com/issues/55347

Details:

1. TEST_recovery_scrub_2: TEST FAILED WITH 1 ERRORS - Ceph - RADOS

2. ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported - Dashboard

3. TestClsRbd.mirror_snapshot failure - Ceph - RBD

4. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

5. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

6. SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure

Remaining 4 Rook test failures are due to repo http://apt.kubernetes.io  is not signed

  - The repository 'https://apt.kubernetes.io kubernetes-xenial InRelease' is not signed.

h3. https://trello.com/c/1LQJnuRh/1759-wip-yuri8-testing-2023-05-23-0802

https://pulpito.ceph.com/?branch=wip-yuri8-testing-2023-05-23-0802

Failures, unrelated:

    1. https://tracker.ceph.com/issues/61225

    2. https://tracker.ceph.com/issues/61402

    3. https://tracker.ceph.com/issues/59678

    4. https://tracker.ceph.com/issues/59057

    5. https://tracker.ceph.com/issues/59380

Details:

    1. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    2. test_dashboard_e2e.sh: AssertionError: Timed out retrying after 120000ms: Expected to find content: '/^smithi160$/' within the selector: 'datatable-body-row datatable-body-cell:nth-child(2)' but never did. - Ceph - Mgr - Dashboard

    3. rados/test_envlibrados_for_rocksdb.sh: Error: Unable to find a match: snappy-devel - Ceph - RADOS

    4. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    5. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

h3. https://trello.com/c/J04nAx3y/1756-wip-yuri11-testing-2023-05-19-0836

https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-05-19-0836

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/61256

    3. https://tracker.ceph.com/issues/59380

    4. https://tracker.ceph.com/issues/58946

    5. https://tracker.ceph.com/issues/61225

    6. https://tracker.ceph.com/issues/59057

    7. https://tracker.ceph.com/issues/58560

    8. https://tracker.ceph.com/issues/57755

    9. https://tracker.ceph.com/issues/55347

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. Upgrade test fails after prometheus_receiver connection is refused - Ceph - Orchestrator

    3. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    4. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    5. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    6. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    7. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    8. task/test_orch_cli: test_cephfs_mirror times out - Ceph - Orchestrator

    9. SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure

h3. https://trello.com/c/nvyHvlZ4/1745-wip-yuri8-testing-2023-05-10-1402

There was an RGW multisite test failure, but it turned out to be related

to an unmerged PR in the batch, which was dropped.

https://pulpito.ceph.com/?branch=wip-yuri8-testing-2023-05-10-1402

https://pulpito.ceph.com/?branch=wip-yuri8-testing-2023-05-18-1232

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58560

    2. https://tracker.ceph.com/issues/59196

    3. https://tracker.ceph.com/issues/61225

    4. https://tracker.ceph.com/issues/58585

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/61256 -- new tracker

    7. https://tracker.ceph.com/issues/59380

    8. https://tracker.ceph.com/issues/59333

Details:

    1. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    2. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    3. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    4. rook: failed to pull kubelet image - Ceph - Orchestrator

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    6. Upgrade test fails after prometheus_receiver connection is refused - Ceph - Orchestrator

    7. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    8. PgScrubber: timeout on reserving replicas - Ceph - RADOS

h3. https://trello.com/c/lM1xjBe0/1744-wip-yuri-testing-2023-05-10-0917

https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-05-10-0917

Failures, unrelated:

    1. https://tracker.ceph.com/issues/61225

    2. https://tracker.ceph.com/issues/58585

    3. https://tracker.ceph.com/issues/44889

    4. https://tracker.ceph.com/issues/59193

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/58560

    7. https://tracker.ceph.com/issues/49287

    8. https://tracker.ceph.com/issues/55347

    9. https://tracker.ceph.com/issues/59380

    10. https://tracker.ceph.com/issues/59192

    11. https://tracker.ceph.com/issues/61261 -- new tracker

    12. https://tracker.ceph.com/issues/61262 -- new tracker

    13. https://tracker.ceph.com/issues/46877

Details:

    1. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    2. rook: failed to pull kubelet image - Ceph - Orchestrator

    3. workunit does not respect suite_branch when it comes to checkout sha1 on remote host - Tools - Teuthology

    4. "Failed to fetch package version from https://shaman.ceph.com/api/search ..." - Infrastructure

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    6. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    7. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator

    8. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    9. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    10. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    11. test_cephadm.sh: missing container image causes job to fail - Ceph - Orchestrator

    12: Cephadm task times out when waiting for osds to come up - Ceph - Orchestrator

    13. mon_clock_skew_check: expected MON_CLOCK_SKEW but got none - Ceph - RADOS

h3. https://trello.com/c/1EFSeXDn/1752-wip-yuri10-testing-2023-05-16-1243

https://pulpito.ceph.com/?branch=wip-yuri10-testing-2023-05-16-1243

Also one failure related to http://archive.ubuntu.com/ubuntu that seems transient.

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/44889

    3. https://tracker.ceph.com/issues/59678

    4. https://tracker.ceph.com/issues/55347

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/61225 -- new tracker

    7. https://tracker.ceph.com/issues/59380

    8. https://tracker.ceph.com/issues/49888

    9. https://tracker.ceph.com/issues/59192

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. workunit does not respect suite_branch when it comes to checkout sha1 on remote host - Tools - Teuthology

    3. rados/test_envlibrados_for_rocksdb.sh: Error: Unable to find a match: snappy-devel - Ceph - RADOS

    4. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    6. TestClsRbd.mirror_snapshot failure - Ceph - RBD

    7. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    8. rados/singleton: radosbench.py: teuthology.exceptions.MaxWhileTries: reached maximum tries (3650) after waiting for 21900 seconds - Ceph - RADOS

    9. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

h3. https://trello.com/c/AjBYBGYC/1738-wip-yuri7-testing-2023-04-19-1343-old-wip-yuri7-testing-2023-04-19-0721-old-wip-yuri7-testing-2023-04-18-0818

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri7-testing-2023-04-19-1343

Failures, unrelated:

    1. https://tracker.ceph.com/issues/57755

    2. https://tracker.ceph.com/issues/58946

    3. https://tracker.ceph.com/issues/49888

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/57754

    6. https://tracker.ceph.com/issues/55347

    7. https://tracker.ceph.com/issues/49287

Details:

    1. task/test_orch_cli: test_cephfs_mirror times out - Ceph - Orchestrator

    2. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    3. rados/singleton: radosbench.py: teuthology.exceptions.MaxWhileTries: reached maximum tries (3650) after waiting for 21900 seconds - Ceph - RADOS

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RGW

    5. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Ceph - RADOS

    6. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    7. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator

h3. https://trello.com/c/YN2r7OyK/1740-wip-yuri3-testing-2023-04-25-1147

https://pulpito.ceph.com/?branch=wip-yuri3-testing-2023-04-25-1147

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59049

    2. https://tracker.ceph.com/issues/59192

    3. https://tracker.ceph.com/issues/59335

    4. https://tracker.ceph.com/issues/59380

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/50371

    7. https://tracker.ceph.com/issues/59057

    8. https://tracker.ceph.com/issues/53345

Details:

    1. WaitReplicas::react(const DigestUpdate&): Unexpected DigestUpdate event - Ceph - RADOS

    2. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)

    3. Found coredumps on smithi related to sqlite3Found coredumps on smithi related to sqlite3 - Ceph - Cephsqlite

    4. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    6. Segmentation fault (core dumped) ceph_test_rados_api_watch_notify_pp - Ceph - RADOS

    7. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    8. Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) - Ceph - Orchestrator

h3. https://trello.com/c/YhSdHR96/1728-wip-yuri2-testing-2023-03-30-0826

https://pulpito.ceph.com/?branch=wip-yuri2-testing-2023-03-30-0826

Failures, unrelated:

    1. https://tracker.ceph.com/issues/51964

    2. https://tracker.ceph.com/issues/58946

    3. https://tracker.ceph.com/issues/58758

    4. https://tracker.ceph.com/issues/58585

    5. https://tracker.ceph.com/issues/59380 -- new tracker

    6. https://tracker.ceph.com/issues/59080

    7. https://tracker.ceph.com/issues/59057

    8. https://tracker.ceph.com/issues/59196

Details:

    1. qa: test_cephfs_mirror_restart_sync_on_blocklist failure - Ceph - CephFS

    2. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    3. qa: fix testcase 'test_cluster_set_user_config_with_non_existing_clusterid' - Ceph - CephFS

    4. rook: failed to pull kubelet image - Ceph - Orchestrator

    5. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

    6. mclock-config.sh: TEST_profile_disallow_builtin_params_modify fails when $res == $opt_val_new - Ceph - RADOS

    7. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    8. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

h3. https://trello.com/c/wCN5TQud/1729-wip-yuri4-testing-2023-03-31-1237

https://pulpito.ceph.com/?branch=wip-yuri4-testing-2023-03-31-1237

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri4-testing-2023-04-07-1825

Failures, unrelated:

    1. https://tracker.ceph.com/issues/17945

    2. https://tracker.ceph.com/issues/59049

    3. https://tracker.ceph.com/issues/59196

    4. https://tracker.ceph.com/issues/56393

    5. https://tracker.ceph.com/issues/58946

    6. https://tracker.ceph.com/issues/49287

    7. https://tracker.ceph.com/issues/55347

    8. https://tracker.ceph.com/issues/59057

    9. https://tracker.ceph.com/issues/59380

Details:

    1. ceph_test_rados_api_tier: failed to decode hitset in HitSetWrite test - Ceph - RADOS

    2. WaitReplicas::react(const DigestUpdate&): Unexpected DigestUpdate event - Ceph - RADOS

    3. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    4. failed to complete snap trimming before timeout - Ceph - RADOS

    5. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    6. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator

    7. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    8. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    9. rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - Ceph - RADOS

h3. https://trello.com/c/8Xlz4rIH/1727-wip-yuri11-testing-2023-03-31-1108-old-wip-yuri11-testing-2023-03-28-0950

https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-03-28-0950

https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-03-31-1108

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58946

    3. https://tracker.ceph.com/issues/58265

    4. https://tracker.ceph.com/issues/59271

    5. https://tracker.ceph.com/issues/59057

    6. https://tracker.ceph.com/issues/59333

    7. https://tracker.ceph.com/issues/59334

    8. https://tracker.ceph.com/issues/59335

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    3. TestClsRbd.group_snap_list_max_read failure during upgrade/parallel tests - Ceph - RBD

    4. mon: FAILED ceph_assert(osdmon()->is_writeable()) - Ceph - RADOS

    5. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    6. PgScrubber: timeout on reserving replicas - Ceph - RADOS

    7. test_pool_create_with_quotas: Timed out after 60s and 0 retries - Ceph - Mgr - Dashboard

    8. Found coredumps on smithi related to sqlite3 - Ceph - Cephsqlite

h3. https://trello.com/c/yauI7omb/1726-wip-yuri7-testing-2023-03-29-1100-old-wip-yuri7-testing-2023-03-28-0942

https://pulpito.ceph.com/?branch=wip-yuri7-testing-2023-03-29-1100

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59192

    2. https://tracker.ceph.com/issues/58585

    3. https://tracker.ceph.com/issues/59057

    4. https://tracker.ceph.com/issues/58946

    5. https://tracker.ceph.com/issues/55347

    6. https://tracker.ceph.com/issues/59196

    7. https://tracker.ceph.com/issues/47838

    8. https://tracker.ceph.com/issues/59080

Details:

    1. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

    2. rook: failed to pull kubelet image - Ceph - Orchestrator

    3. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    4. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    5. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    6. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    7. mon/test_mon_osdmap_prune.sh: first_pinned != trim_to - Ceph - RADOS

    8. mclock-config.sh: TEST_profile_disallow_builtin_params_modify fails when $res == $opt_val_new - Ceph - RADOS

h3. https://trello.com/c/epwSlEHP/1722-wip-yuri4-testing-2023-03-25-0714-old-wip-yuri4-testing-2023-03-24-0910

https://pulpito.ceph.com/?branch=wip-yuri4-testing-2023-03-25-0714

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/59196

    3. https://tracker.ceph.com/issues/59271

    4. https://tracker.ceph.com/issues/58560

    5. https://tracker.ceph.com/issues/51964

    6. https://tracker.ceph.com/issues/58560

    7. https://tracker.ceph.com/issues/59192

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    2. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    3. mon: FAILED ceph_assert(osdmon()->is_writeable()) - Ceph - RADOS

    4. rook: failed to pull kubelet image - Ceph - Orchestrator

    5. qa: test_cephfs_mirror_restart_sync_on_blocklist failure - Ceph  - CephFS

    6. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Ceph - RADOS

    7. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

h3. https://trello.com/c/PEo71l9g/1720-wip-aclamk-bs-elastic-shared-blob-save-25032023-a-old-wip-lflores-testing-2023-03-22-2113

http://pulpito.front.sepia.ceph.com/?branch=wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a

Failures, unrelated:

    1. https://tracker.ceph.com/issues/59058

    2. https://tracker.ceph.com/issues/56034

    3. https://tracker.ceph.com/issues/58585

    4. https://tracker.ceph.com/issues/59172

    5. https://tracker.ceph.com/issues/56192

    6. https://tracker.ceph.com/issues/49287

    7. https://tracker.ceph.com/issues/58758

    8. https://tracker.ceph.com/issues/58946

    9. https://tracker.ceph.com/issues/59057

    10. https://tracker.ceph.com/issues/59192

Details:

    1. ceph_test_lazy_omap_stats segfault while waiting for active+clean - Ceph - RADOS

    2. qa/standalone/osd/divergent-priors.sh fails in test TEST_divergent_3() - Ceph - RADOS

    3. rook: failed to pull kubelet image - Ceph - Orchestrator

    4. test_pool_min_size: AssertionError: wait_for_clean: failed before timeout expired due to down PGs - Ceph - RADOS

    5. crash: virtual Monitor::~Monitor(): assert(session_map.sessions.empty()) - Ceph - RADOS

    6. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found

    7. qa: fix testcase 'test_cluster_set_user_config_with_non_existing_clusterid' - Ceph - CephFS

    8. cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

    9. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

    10. cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS

h3. https://trello.com/c/Qa8vTuf8/1717-wip-yuri4-testing-2023-03-15-1418

https://pulpito.ceph.com/?branch=wip-yuri4-testing-2023-03-15-1418

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/56393

    3. https://tracker.ceph.com/issues/59123

    4. https://tracker.ceph.com/issues/58585

    5. https://tracker.ceph.com/issues/58560

    6. https://tracker.ceph.com/issues/59127

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    2. thrash-erasure-code-big: failed to complete snap trimming before timeout - Ceph - RADOS

    3. Timeout opening channel - Infrastructure

    4. rook: failed to pull kubelet image - Ceph - Orchestrator

    5. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    6. Job that normally complete much sooner last almost 12 hours - Infrastructure

h3. https://trello.com/c/fo5GZ0YC/1712-wip-yuri7-testing-2023-03-10-0830

https://pulpito.ceph.com/?branch=wip-yuri7-testing-2023-03-10-0830

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/59079

    3. https://tracker.ceph.com/issues/59080

    4. https://tracker.ceph.com/issues/58585

    6. https://tracker.ceph.com/issues/59057

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    2. AssertionError: timeout expired in wait_for_all_osds_up - Ceph - RADOS

    3. mclock-config.sh: TEST_profile_disallow_builtin_params_modify fails when $res == $opt_val_new - Ceph - RADOS

    4. rook: failed to pull kubelet image - Ceph - Orchestrator

    5. rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_test' on centos 8 - Ceph - RADOS

h3. https://trello.com/c/EbLKJDPm/1685-wip-yuri11-testing-2023-03-08-1220-old-wip-yuri11-testing-2023-03-01-1424-old-wip-yuri11-testing-2023-02-20-1329-old-wip-yuri11

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri11-testing-2023-03-08-1220

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58560

    3. https://tracker.ceph.com/issues/58946

    4. https://tracker.ceph.com/issues/49287

    5. https://tracker.ceph.com/issues/57755

    6. https://tracker.ceph.com/issues/52316

    7. https://tracker.ceph.com/issues/58496

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    3. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    4. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    5. task/test_orch_cli: test_cephfs_mirror times out - Ceph - Orchestrator

    6. qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) - Ceph - RADOS

    7. osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty()) - Ceph - RADOS

h3. https://trello.com/c/u5ydxGCS/1698-wip-yuri7-testing-2023-02-27-1105

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri7-testing-2023-02-27-1105

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58475

    3. https://tracker.ceph.com/issues/57754

    4. https://tracker.ceph.com/issues/50786

    5. https://tracker.ceph.com/issues/49287

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    3. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Ceph - RADOS

    4. UnicodeDecodeError: 'utf8' codec can't decode byte - Ceph - RADOS

    5. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

h3. https://trello.com/c/hIlO2MJn/1706-wip-yuri8-testing-2023-03-07-1527

https://pulpito.ceph.com/?branch=wip-yuri8-testing-2023-03-07-1527

Failures, unrelated:

    1. https://tracker.ceph.com/issues/49287

    2. https://tracker.ceph.com/issues/58585

    3. https://tracker.ceph.com/issues/58560

    4. https://tracker.ceph.com/issues/58946

    5. https://tracker.ceph.com/issues/51964

Details:

    1. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    2. rook: failed to pull kubelet image - Ceph - Orchestrator

    3. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    4. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    5. qa: test_cephfs_mirror_restart_sync_on_blocklist failure - Ceph - CephFS

h3. https://trello.com/c/bLUA7Wf5/1705-wip-yuri4-testing-2023-03-08-1234-old-wip-yuri4-testing-2023-03-07-1351

https://pulpito.ceph.com/?branch=wip-yuri4-testing-2023-03-08-1234

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58946

    2. https://tracker.ceph.com/issues/58560

    3. https://tracker.ceph.com/issues/58585

    4. https://tracker.ceph.com/issues/55347

    5. https://tracker.ceph.com/issues/49287

Details:

    1. cephadm: KeyError: 'osdspec_affinity' - Ceph - Orchestrator

    2. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    3. rook: failed to pull kubelet image - Ceph - Orchestrator

    4. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    5. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator

h3. Main baseline 2/24/23

https://pulpito.ceph.com/?sha1=f9d812a56231a14fafcdfb339f87d3d9a9e6e55f

Failures:

    1. https://tracker.ceph.com/issues/58560

    2. https://tracker.ceph.com/issues/57771

    3. https://tracker.ceph.com/issues/58585

    4. https://tracker.ceph.com/issues/58475

    5. https://tracker.ceph.com/issues/58758

    6. https://tracker.ceph.com/issues/58797

    7. https://tracker.ceph.com/issues/58893 -- new tracker

    8. https://tracker.ceph.com/issues/49428

    9. https://tracker.ceph.com/issues/55347

    10. https://tracker.ceph.com/issues/58800

Details:

    1. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    2. orch/cephadm suite: 'TESTDIR=/home/ubuntu/cephtest bash -s' fails - Ceph - Orchestrator

    3. rook: failed to pull kubelet image - Ceph - Orchestrator

    4. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21

    5. qa: fix testcase 'test_cluster_set_user_config_with_non_existing_clusterid' - Ceph - CephFS

    6. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test"

    7. test_map_discontinuity: AssertionError: wait_for_clean: failed before timeout expired - Ceph - RADOS

    8. ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" - Ceph - RADOS

    9. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator

    10. ansible: Failed to update apt cache: unknown reason - Infrastructure - Sepia

h3. https://trello.com/c/t4cQVOvQ/1695-wip-yuri10-testing-2023-02-22-0848

http://pulpito.front.sepia.ceph.com:80/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi

http://pulpito.front.sepia.ceph.com:80/yuriw-2023-02-23_16:14:52-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi

Failures, unrelated:

    1. https://tracker.ceph.com/issues/57754

    2. https://tracker.ceph.com/issues/58475

    3. https://tracker.ceph.com/issues/58585

    4. https://tracker.ceph.com/issues/58797

Details:

    1. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Ceph - RADOS

    2. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    3. rook: failed to pull kubelet image - Ceph - Orchestrator

    4. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test"

h3. https://trello.com/c/hrTt8qIn/1693-wip-yuri6-testing-2023-02-24-0805-old-wip-yuri6-testing-2023-02-21-1406

https://pulpito.ceph.com/?branch=wip-yuri6-testing-2023-02-24-0805

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58560

    3. https://tracker.ceph.com/issues/58797

    4. https://tracker.ceph.com/issues/58744

    5. https://tracker.ceph.com/issues/58475

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    3. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test" - Ceph - RADOS

    4. qa: intermittent nfs test failures at nfs cluster creation - Ceph - CephFS

    5. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

h3. https://trello.com/c/gleu2p6U/1689-wip-yuri-testing-2023-02-22-2037-old-wip-yuri-testing-2023-02-16-0839

https://pulpito.ceph.com/yuriw-2023-02-16_22:44:43-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi

https://pulpito.ceph.com/lflores-2023-02-20_21:22:20-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi

https://pulpito.ceph.com/yuriw-2023-02-23_16:42:54-rados-wip-yuri-testing-2023-02-22-2037-distro-default-smithi

https://pulpito.ceph.com/lflores-2023-02-23_17:54:36-rados-wip-yuri-testing-2023-02-22-2037-distro-default-smithi

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58560

    3. https://tracker.ceph.com/issues/58496

    4. https://tracker.ceph.com/issues/49961

    5. https://tracker.ceph.com/issues/58861

    6. https://tracker.ceph.com/issues/58797

    7. https://tracker.ceph.com/issues/49428

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure

    3. osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty()) - Ceph - RADOS

    4. scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed - Ceph - RADOS

    5. OSError: cephadm config file not found - Ceph - Orchestrator

    6. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test" - Ceph - RADOS

    7. ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" - Ceph - RADOS

h3. https://trello.com/c/FzMz7O3S/1683-wip-yuri10-testing-2023-02-15-1245-old-wip-yuri10-testing-2023-02-06-0846-old-wip-yuri10-testing-2023-02-06-0809

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri10-testing-2023-02-15-1245

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585

    2. https://tracker.ceph.com/issues/58475

    3. https://tracker.ceph.com/issues/58797 -- new tracker; seen in main baseline, therefore unrelated to trackers in this batch

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    3. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test" - Ceph - RADOS

h3. https://trello.com/c/buZUPZx0/1680-wip-yuri2-testing-2023-02-08-1429-old-wip-yuri2-testing-2023-02-06-1140-old-wip-yuri2-testing-2023-01-26-1532

https://pulpito.ceph.com/?branch=wip-yuri2-testing-2023-01-26-1532

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58496 -- fix in progress

    2. https://tracker.ceph.com/issues/58585

    3. https://tracker.ceph.com/issues/58475

    4. https://tracker.ceph.com/issues/57754

    5. https://tracker.ceph.com/issues/49287

    6. https://tracker.ceph.com/issues/57731

    7. https://tracker.ceph.com/issues/54829

    8. https://tracker.ceph.com/issues/52221

Details:

    1. osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty()) - Ceph - RADOS

    2. rook: failed to pull kubelet image - Ceph - Orchestrator

    3. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    4. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure

    5. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator

    6. Problem: package container-selinux conflicts with udica < 0.2.6-1 provided by udica-0.2.4-1 - Infrastructure

    7. crash: void OSDMap::check_health(ceph::common::CephContext*, health_check_map_t*) const: assert(num_down_in_osds <= num_in_osds) - Ceph - RADOS

    8. crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end()) - Ceph - RADOS

h3. https://trello.com/c/GA6hud1j/1674-wip-yuri-testing-2023-01-23-0926-old-wip-yuri-testing-2023-01-12-0816-old-wip-yuri-testing-2023-01-11-0818-old-wip-yuri-testing

https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-01-23-0926

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58587 -- new tracker

    2. https://tracker.ceph.com/issues/58585

    3. https://tracker.ceph.com/issues/58098 -- fix merged to latest main

    4. https://tracker.ceph.com/issues/58256 -- fix merged to latest main

    5. https://tracker.ceph.com/issues/57900

    6. https://tracker.ceph.com/issues/58475

    7. https://tracker.ceph.com/issues/58560

Details:

    1. test_dedup_tool.sh: test_dedup_object fails when pool 'dedup_chunk_pool' does not exist - Ceph - RADOS

    2. rook: failed to pull kubelet image - Ceph - Orchestrator

    3. qa/workunits/rados/test_crash.sh: crashes are never posted - Ceph - RADOS

    4. ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2: Expected: (logger->get(l_bluefs_slow_used_bytes)) >= (16 * 1024 * 1024), actual: 0 vs 16777216 - Ceph - RADOS

    5. mon/crush_ops.sh: mons out of quorum - Ceph - RADOS

    6. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    7. test_envlibrados_for_rocksdb.sh failed to subscrib repo - Ceph

h3. https://trello.com/c/583LyrTc/1667-wip-yuri2-testing-2023-01-23-0928-old-wip-yuri2-testing-2023-01-12-0816-old-wip-yuri2-testing-2023-01-11-0819-old-wip-yuri2-test

https://pulpito.ceph.com/?branch=wip-yuri2-testing-2023-01-23-0928

Failures, unrelated:

    1. https://tracker.ceph.com/issues/58585 -- new tracker

    2. https://tracker.ceph.com/issues/58256 -- fix merged to latest main

    3. https://tracker.ceph.com/issues/58475

    4. https://tracker.ceph.com/issues/57754 -- closed

    5. https://tracker.ceph.com/issues/57546 -- fix is in testing

Details:

    1. rook: failed to pull kubelet image - Ceph - Orchestrator

    2. ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2: Expected: (logger->get(l_bluefs_slow_used_bytes)) >= (16 * 1024 * 1024), actual: 0 vs 16777216 - Ceph - RADOS

    3. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard

    4. test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Ceph - RADOS

    5. rados/thrash-erasure-code: wait_for_recovery timeout due to "active+clean+remapped+laggy" pgs - Ceph - RADOS

h3. main baseline review -- https://pulpito.ceph.com/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/

Failures:

    1. https://tracker.ceph.com/issues/58098 -- fix is in testing; holdup is issues with rhel satellite

    2. https://tracker.ceph.com/issues/58258

    3. https://tracker.ceph.com/issues/56000

    4. https://tracker.ceph.com/issues/57632 -- fix is awaiting a review from the core team

    5. https://tracker.ceph.com/issues/58475 -- new tracker

    6. https://tracker.ceph.com/issues/57731

    7. https://tracker.ceph.com/issues/58476 -- new tracker

    8. https://tracker.ceph.com/issues/57303

    9. https://tracker.ceph.com/issues/58256 -- fix is in testing

    10. https://tracker.ceph.com/issues/57546 -- fix is in testing

    11. https://tracker.ceph.com/issues/58496 -- new tracker

Details:

    1. qa/workunits/rados/test_crash.sh: crashes are never posted - Ceph - RADOS

    2. rook: kubelet fails from connection refused - Ceph - Orchestrator

    3. task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls` - Ceph - Orchestrator