Quincy¶
On-call Schedule¶
- Feb: Venky
- Mar: Patrick
- Apr: Jos
- May: Xiubo
- Jun: Rishabh
- Jul: Kotresh
- Aug: Milind
- Sep: Leonid
- Oct: Dhairya
- Nov: Chris
2024 March 26¶
https://tracker.ceph.com/issues/65134
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-quincy-2024-03-14-0655-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/65147
quincy: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) - https://tracker.ceph.com/issues/64711
Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/61892
test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/54606
check-counter task runs till max job timeout
2024 Jan 31¶
- https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - https://tracker.ceph.com/issues/59534
qa/workunits/suites/dbench.sh failed with "write failed on handle 9938 (Input/output error)" - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/58476
test_non_existent_cluster: cluster does not exist - Ceph - CephFS - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the
2024 Jan 17¶
- https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/58476
test_non_existent_cluster: cluster does not exist - Ceph - CephFS - https://tracker.ceph.com/issues/64059
ior.tbz2 not found (new) - https://tracker.ceph.com/issues/64060
Test failure: test_subvolume_group_rm_when_its_not_empty (tasks.cephfs.test_volumes.TestSubvolumeGroups) (new) - https://tracker.ceph.com/issues/61892
test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke)
2024 Jan 12¶
- https://tracker.ceph.com/issues/58476
test_non_existent_cluster: cluster does not exist - Ceph - CephFS - https://tracker.ceph.com/issues/64011 (new)
qa: Command failed qa/workunits/suites/pjd.sh - https://tracker.ceph.com/issues/61892
test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/64012 (new)
qa: Command failed qa/workunits/fs/full/subvolume_clone.sh
2024 Jan 2¶
- https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/58476
test_non_existent_cluster: cluster does not exist - Ceph - CephFS - https://tracker.ceph.com/issues/63931
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/63212
qa: failed to download ior.tbz2 - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023 Dec 27¶
- https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/63212
qa: failed to download ior.tbz2 - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/63894
orchestrator: cephadm failed - alertmanager container not found
2023 Dec 21¶
https://pulpito.ceph.com/?branch=wip-yuri11-testing-2023-12-14-1108-quincy
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/63931
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023 Dec 20¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-2023-12-18-1207-reef-2
(Lots of centos/rhel related issues)
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/50224
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/57087
qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS
2023 December 14¶
- https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/50224
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh
2023 October 19¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-quincy-20231019.172112
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/62278 (missed qa fix in back port)
pybind/mgr/volumes: pending_subvolume_deletions count is always zero in fs volume info output - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/61399
qa: build failure for ior (tarball name changed, so test fails with missing tarball - https://tracker.ceph.com/issues/61399#note-20) - https://tracker.ceph.com/issues/62658
error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash
2023 October 10¶
https://pulpito.ceph.com/?branch=wip-yuri3-testing-2023-10-10-0720-quincy
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/61399
qa: build failure for ior (tarball name changed, so test fails with missing tarball - https://tracker.ceph.com/issues/61399#note-20) - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/62278 (missed qa fix in back port)
pybind/mgr/volumes: pending_subvolume_deletions count is always zero in fs volume info output - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/57255
rados/cephadm/mds_upgrade_sequence, pacific : cephadm [ERR] Upgrade: Paused due to UPGRADE_NO_STANDBY_MGR: Upgrade: Need standby mgr daemon
2023 October 09¶
https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-10-06-0949-quincy
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/61182
qa: workloads/cephfs-mirror-ha-workunit - stopping mirror daemon after the test finishes timesout. - https://tracker.ceph.com/issues/61399
qa: build failure for ior (tarball name changed, so test fails with missing tarball - https://tracker.ceph.com/issues/61399#note-20)
2023 October 06¶
https://pulpito.ceph.com/?branch=wip-yuri3-testing-2023-10-06-0948-quincy
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/63132
qa: subvolume_snapshot_rm.sh stalls when waiting for OSD_FULL warning - https://tracker.ceph.com/issues/62810
Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again - https://tracker.ceph.com/issues/59343
qa: fs/snaps/snaptest-multiple-capsnaps.sh failed (pending kclient fix) - https://tracker.ceph.com/issues/61892
[testing] qa: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023 October 03¶
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the - https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - https://tracker.ceph.com/issues/63071
qa: Test failure: test_valid_dump_blocked_ops_count (tasks.cephfs.test_admin.TestValidTell) - https://tracker.ceph.com/issues/61394
mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log
2023 August 08¶
https://trello.com/c/ZjPC9CcN/1820-wip-yuri5-testing-2023-08-08-0807-quincy
https://pulpito.ceph.com/?branch=wip-yuri5-testing-2023-08-08-0807-quincy
- https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/62484
quincy (?): qa: ffsb.sh test failure - https://tracker.ceph.com/issues/62485
quincy (?): pybind/mgr/volumes: subvolume rm timeout - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/62489
testing: did not reconnect to MDS during up:reconnect
4 August 2023¶
https://pulpito.ceph.com/?branch=wip-yuri7-testing-2023-07-27-1336-quincy
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
25 July 2023¶
https://pulpito.ceph.com/?branch=wip-yuri3-testing-2023-07-14-0724-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/61775
cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests) - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh
2023 July 04¶
https://pulpito.ceph.com/yuriw-2023-07-03_15:34:02-fs-quincy_release-distro-default-smithi/
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/61775
cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests) - https://tracker.ceph.com/issues/61892
Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023 June 14¶
- https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - Failed to fetch package version
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-13_23:20:02-fs-wip-yuri3-testing-2023-06-13-1204-quincy-distro-default-smithi/7303252
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-13_23:20:02-fs-wip-yuri3-testing-2023-06-13-1204-quincy-distro-default-smithi/7303360 - cephfs_mirror: reached maximum tries (51) after waiting for 300 seconds
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-13_23:20:02-fs-wip-yuri3-testing-2023-06-13-1204-quincy-distro-default-smithi/7303322
2023 June 07¶
- https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/61609
CommandFailedError for qa/workunits/libcephfs/test.sh - https://tracker.ceph.com/issues/61610
CommandFailedError for qa/workunits/suites/fsstress.sh - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/61182
workloads/cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds (mirror daemon stop times out) - https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
- http://pulpito.front.sepia.ceph.com/yuriw-2023-05-31_21:56:15-fs-wip-yuri6-testing-2023-05-31-0933-quincy-distro-default-smithi/7292615
- http://pulpito.front.sepia.ceph.com/yuriw-2023-05-31_21:56:15-fs-wip-yuri6-testing-2023-05-31-0933-quincy-distro-default-smithi/7292784
2023 May 24¶
- https://tracker.ceph.com/issues/61393 (NEW - not related)
orchestrator bug: cephadm command failed - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/55332
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/61394 (NEW - not related)
mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log - https://tracker.ceph.com/issues/61182
workloads/cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds (mirror daemon stop times out) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark
tools (e.g. Fio)" - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - Failed to fetch package version
https://pulpito.ceph.com/yuriw-2023-05-23_15:23:11-fs-wip-yuri10-testing-2023-05-18-0815-quincy-distro-default-smithi/7284063
https://pulpito.ceph.com/yuriw-2023-05-23_15:23:11-fs-wip-yuri10-testing-2023-05-18-0815-quincy-distro-default-smithi/7284130
2023 Apr 21/24¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20230420.183701-quincy
2 Failures:
"Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=58e06d348d8a2da339540be5425a40ec7683e512 "
are a side-effect of revert https://github.com/ceph/ceph/pull/51029 . This is expected and should be fixed by the new backport that is reverted.
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/54460
snaptest-multiple-capsnaps.sh test failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/59532
quincy: cephadm.upgrade from 16.2.4 (related?) stuck with one OSD upgraded
2023 Mar 02¶
https://pulpito.ceph.com/yuriw-2023-02-22_20:50:58-fs-wip-yuri4-testing-2023-02-22-0817-quincy-distro-default-smithi/
https://pulpito.ceph.com/yuriw-2023-02-28_22:41:58-fs-wip-yuri10-testing-2023-02-28-0752-quincy-distro-default-smithi/
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/58726
Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds
2023 Feb 17¶
- https://tracker.ceph.com/issues/58754
Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/58756
qa: error during scrub thrashing - https://tracker.ceph.com/issues/58757
qa: Command failed (workunit test suites/fsstress.sh)
2023 Feb 16¶
- https://tracker.ceph.com/issues/58746
qa: VersionNotFoundError: Failed to fetch package version - https://tracker.ceph.com/issues/58745
qa: cephadm failed to stop mon
2023 Feb 15¶
- https://tracker.ceph.com/issues/57446
Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/58656
qa: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/58726
quincy: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) - https://tracker.ceph.com/issues/58727
quincy: Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays)
2023 Feb 07¶
- https://tracker.ceph.com/issues/58656
qa: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2022 Oct 21¶
- https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57446
Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022 Oct 17¶
- https://tracker.ceph.com/issues/54460
snaptest-multiple-capsnaps.sh test failure - https://tracker.ceph.com/issues/57446
Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/55825
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
cluster [WRN] client.4490 isn't responding to mclientcaps(revoke)
2022 Sep 29¶
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri6-testing-2022-09-23-1008-quincy
- https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57446
qa: test_subvolume_snapshot_info_if_orphan_clone fails - https://tracker.ceph.com/issues/50224
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/57280
qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke)
2022 Sep 09¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57446
qa: test_subvolume_snapshot_info_if_orphan_clone fails - https://tracker.ceph.com/issues/51964
Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/57280
Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing
2022 Sep 02¶
and
https://pulpito.ceph.com/?branch=wip-lflores-testing-2-2022-08-26-2240-quincy
- https://tracker.ceph.com/issues/57280
Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/48773
error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
2022 Aug 31¶
https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-08-23-1120-quincy
- https://tracker.ceph.com/issues/51964
Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/57280
Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/48773
error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke)
2022 Aug 17¶
There were following errors not related to tests which fixed in rerun:
- Command failed on smithi161 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell
c ... -bash -c 'ceph fs dump'" - Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing
- reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up
- https://tracker.ceph.com/issues/56697 - qa: fs/snaps fails for fuse - Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi150 with status 1: 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp ..." - SSH connection to smithi077 was lost: 'sudo rm
rf -/home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
2022 Aug 10¶
http://pulpito.front.sepia.ceph.com/yuriw-2022-08-11_02:21:28-fs-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/- Most of the failures are passed in re-run. Please check rerun failures below.
- tasks/{1-thrash/mon 2-workunit/fs/snaps - reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up
- tasks/{1-thrash/osd 2-workunit/suites/iozone - reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up
- tasks/metrics - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
- tasks/scrub - No module named 'tasks.cephfs.fuse_mount'
- tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} - No module named 'tasks.fs'
- tasks/snap-schedule - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
- tasks/volumes/{overrides test/clone}} - No module named 'tasks.ceph'
- tasks/snapshots - CommandFailedError: Command failed on smithi035 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes - INFO:teuthology.orchestra.run.smithi035.stderr:E: Version '17.2.3-414-ge5c30ac2-1focal' for 'python-ceph' was not found - INFO:teuthology.orchestra.run.smithi035.stderr:E: Unable to locate package libcephfs1
- tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} - No module named 'tasks.ceph'
- tasks/{1-thrash/osd 2-workunit/suites/pjd}} - No module named 'tasks.ceph'
- tasks/cfuse_workunit_suites_fsstress traceless/50pc} - No module named 'tasks'
- tasks/{0-octopus 1-upgrade}} - No module named 'tasks'
- tasks/{1-thrash/osd 2-workunit/fs/snaps}} - cluster [WRN] client.4520 isn't responding to mclientcaps(revoke),
- tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} - reached maximum tries (90) after waiting for 540 seconds - teuthology.misc:7 of 8 OSDs are up
- tasks/{1-thrash/mon 2-workunit/fs/snaps - reached maximum tries (90) after waiting for 540 seconds
DEBUG:teuthology.misc:7 of 8 OSDs are up - http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) - tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} - reached maximum tries (90) after waiting for 540 seconds
DEBUG:teuthology.misc:7 of 8 OSDs are up
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022 Aug 03¶
https://pulpito.ceph.com/yuriw-2022-08-04_11:54:20-fs-wip-yuri8-testing-2022-08-03-1028-quincy-distro-default-smithi/
Re-run: https://pulpito.ceph.com/yuriw-2022-08-09_15:36:21-fs-wip-yuri8-testing-2022-08-03-1028-quincy-distro-default-smithi
- No module named 'tasks' - Fixed in re-run
- https://tracker.ceph.com/issues/51282
cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
- https://tracker.ceph.com/issues/57064
qa: test_add_ancestor_and_child_directory failure
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
- https://tracker.ceph.com/issues/50223
cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke)
2022 Jul 22¶
https://pulpito.ceph.com/yuriw-2022-07-11_13:37:40-fs-wip-yuri5-testing-2022-07-06-1020-quincy-distro-default-smithi/
re-run: https://pulpito.ceph.com/yuriw-2022-07-12_13:37:44-fs-wip-yuri5-testing-2022-07-06-1020-quincy-distro-default-smithi/
Most failure weren't seen in re-run.
- http://tracker.ceph.com/issues/52624
Health check failed: Reduced data availability - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
2022 Jul 13¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/51964
Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/48773
error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'
2022 Jun 08¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022 Jun 07¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022 Jun 03¶
https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-06-02-0810-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
2022 May 31¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022 May 26¶
https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-05-10-1027-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
2022 May 10¶
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri-testing-2022-05-05-0838-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)"
2022 April 29¶
https://pulpito.ceph.com/?branch=wip-yuri3-testing-2022-04-22-0534-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
2022 April 13¶
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri3-testing-2022-04-11-0746-quincy
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout
2022 March 31¶
http://pulpito.front.sepia.ceph.com/yuriw-2022-03-29_20:09:22-fs-wip-yuri-testing-2022-03-29-0741-quincy-distro-default-smithi/
http://pulpito.front.sepia.ceph.com/yuriw-2022-03-30_14:35:58-fs-wip-yuri-testing-2022-03-29-0741-quincy-distro-default-smithi/
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/54460
snaptest-multiple-capsnaps.sh test failure - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - http://tracker.ceph.com/issues/54606
check-counter task runs till max job timeout
Handful of failed jobs due to:
Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c5bb4e7d582f118c1093d94fbfedfb197eaa03b4 -v bootstrap --fsid 44e07f86-b03b-11ec-8c35-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.55 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'
2022 March 17¶
- http://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - http://tracker.ceph.com/issues/54461
ffsb.sh test failure - http://tracker.ceph.com/issues/54606
check-counter task runs till max job timeout
Couple of jobs that are dead with:
2022-03-15T05:15:22.447 ERROR:paramiko.transport:Socket exception: No route to host (113) 2022-03-15T05:15:22.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-03-15T05:15:22.453 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0...
2022 March 1¶
- https://tracker.ceph.com/issues/51282 (maybe?)
cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/52624
cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi152 with status 1: 'mkdirp -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' - https://tracker.ceph.com/issues/50223
cluster [WRN] client.14480 isn't responding to mclientcaps(revoke), ino 0x1000000f3fd pending pAsLsXsFsc issued pAsLsXsFscb, sent 304.933510 seconds ago" in cluster log - https://tracker.ceph.com/issues/54461
Command failed (workunit test suites/ffsb.sh) on smithi124 with status 1: 'mkdirp -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128: 'mkdirp -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'