Main » History » Revision 247
Revision 246 (Rishabh Dave, 04/04/2024 08:04 AM) → Revision 247/279 (Rishabh Dave, 04/04/2024 08:04 AM)
h1. <code>main</code> branch h3. ADD NEW ENTRY HERE h3. 4 Apr 2024 TEMP https://pulpito.ceph.com/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/ * https://tracker.ceph.com/issues/64927 qa/cephfs: test_cephfs_mirror_blocklist raises "KeyError: 'rados_inst'" * https://tracker.ceph.com/issues/65022 qa: test_max_items_per_obj open procs not fully cleaned up * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure * https://tracker.ceph.com/issues/65136 QA failure: test_fscrypt_dummy_encryption_with_quick_group * https://tracker.ceph.com/issues/65246 qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize) * https://tracker.ceph.com/issues/58945 qa: xfstests-dev's generic test suite has failures with fuse client * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/63265 qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/48562 qa: scrub - object missing on disk; some files may be lost * https://tracker.ceph.com/issues/65020 qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log * https://tracker.ceph.com/issues/64572 workunits/fsx.sh failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/64502 client: ceph-fuse fails to unmount after upgrade to main * https://tracker.ceph.com/issues/65018 PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)" * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/54741 crash: MDSTableClient::got_journaled_ack(unsigned long) h3. 2024-04-02 https://tracker.ceph.com/issues/65215 * "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676 * "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972 * "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502 * "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711 * "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572 * "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699 * "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774 * "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022 * "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220 * "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707 * "qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log":https://tracker.ceph.com/issues/65021 * "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020 * "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108 * "ffsb.sh failure Resource temporarily unavailable":https://tracker.ceph.com/issues/62067 * "QA failure: test_fscrypt_dummy_encryption_with_quick_group":https://tracker.ceph.com/issues/65136 * "qa: cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled in cluster log":https://tracker.ceph.com/issues/65271 * "qa: test_cephfs_mirror_cancel_sync fails in a 100 jobs run of fs:mirror suite":https://tracker.ceph.com/issues/64534 h3. 2024-03-28 https://tracker.ceph.com/issues/65213 * "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676 * "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572 * "PG_DEGRADED warnings during cluster creation via cephadm: Health check failed: Degraded data":https://tracker.ceph.com/issues/65018 * "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707 * "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972 * "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108 * "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020 * "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699 * "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711 * "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022 * "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502 * "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774 * "qa: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)":https://tracker.ceph.com/issues/52624 * "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220 h3. 2024-03-25 https://pulpito.ceph.com/mchangir-2024-03-22_09:46:06-fs:upgrade-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/ * https://tracker.ceph.com/issues/64502 fusermount -u fails with: teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/ * https://tracker.ceph.com/issues/62245 libcephfs/test.sh failed - https://tracker.ceph.com/issues/62245#note-3 h3. 2024-03-20 https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742 https://github.com/batrick/ceph/commit/360516069d9393362c4cc6eb9371680fe16d66ab Ubuntu jobs filtered out because builds were skipped by jenkins/shaman. This run has a lot more failures because https://github.com/ceph/ceph/pull/55455 fixed log WRN/ERR checks. * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/64572 workunits/fsx.sh failure * https://tracker.ceph.com/issues/65018 PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)" * https://tracker.ceph.com/issues/64707 (new issue) suites/fsstress.sh hangs on one client - test times out * https://tracker.ceph.com/issues/64988 qa: fs:workloads mgr client evicted indicated by "cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds" * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/64972 qa: "ceph tell 4.3a deep-scrub" command not found * https://tracker.ceph.com/issues/54108 qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" * https://tracker.ceph.com/issues/65019 qa/suites/fs/top: [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log * https://tracker.ceph.com/issues/65020 qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log * https://tracker.ceph.com/issues/65021 qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/64711 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/65022 qa: test_max_items_per_obj open procs not fully cleaned up h3. 14th March 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240307.013758 (pjd.sh failures are related to a bug in the testing kernel. See - https://tracker.ceph.com/issues/64679#note-4) * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/64502 pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main * https://tracker.ceph.com/issues/64572 workunits/fsx.sh failure * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed h3. 5th March 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240304.042522 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/64502 pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/64572 workunits/fsx.sh failure * https://tracker.ceph.com/issues/64707 (new issue) suites/fsstress.sh hangs on one client - test times out * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure * https://tracker.ceph.com/issues/64711 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/64729 (new issue) mon.a (mon.0) 1281 : cluster 3 [WRN] MDS_SLOW_METADATA_IO: 3 MDSs report slow metadata IOs" in cluster log * https://tracker.ceph.com/issues/64730 fs/misc/multiple_rsync.sh workunit times out h3. 26th Feb 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240216.060239 (This run is a bit messy due to a) OCI runtime issues in the testing kernel with centos9 b) SELinux denials related failures c) Unrelated MON_DOWN warnings) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/64172 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/64572 workunits/fsx.sh failure h3. 20th Feb 2024 https://github.com/ceph/ceph/pull/55601 https://github.com/ceph/ceph/pull/55659 https://pulpito.ceph.com/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/ * https://tracker.ceph.com/issues/64502 client: quincy ceph-fuse fails to unmount after upgrade to main This run has numerous problems. #55601 introduces testing for the upgrade sequence from </code>reef/{v18.2.0,v18.2.1,reef}</code> as well as an extra dimension for the ceph-fuse client. The main "big" issue is i64502: the ceph-fuse client is not being unmounted when <code>fusermount -u</code> is called. Instead, the client begins to unmount only after daemons are shut down during test cleanup. h3. 19th Feb 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240217.015652 * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/63764 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/64482 ceph: stderr Error: OCI runtime error: crun: bpf create ``: Function not implemented h3. 29 Jan 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240119.075157-1 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/64172 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) * https://tracker.ceph.com/issues/63265 qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/59684 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/64209 snaptest-multiple-capsnaps.sh fails with "got remote process result: 1" h3. 17th Jan 2024 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240103.072409-1 * https://tracker.ceph.com/issues/63764 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/63949 leak in mds.c detected by valgrind during CephFS QA run * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/63259 mds: failed to store backtrace and force file system read-only * https://tracker.ceph.com/issues/63265 qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' h3. 16 Jan 2024 https://pulpito.ceph.com/rishabh-2023-12-11_15:37:57-fs-rishabh-2023dec11-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-12-17_11:19:43-fs-rishabh-2023dec11-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2024-01-04_18:43:16-fs-rishabh-2024jan4-testing-default-smithi * https://tracker.ceph.com/issues/63764 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63949 valgrind leak in MDS * https://tracker.ceph.com/issues/64041 qa/cephfs: fs/upgrade/nofs suite attempts to jump more than 2 releases * fsstress failure in last run was due a kernel MM layer failure, unrelated to CephFS * from last run, job #7507400 failed due to MGR; FS wasn't degraded, so it's unrelated to CephFS h3. 06 Dec 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818-x (rerun w/ squid kickoff changes) * https://tracker.ceph.com/issues/63764 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/62580 testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout * https://tracker.ceph.com/issues/63265 qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' * https://tracker.ceph.com/issues/63806 ffsb.sh workunit failure (MDS: std::out_of_range, damaged) h3. 30 Nov 2023 https://pulpito.ceph.com/pdonnell-2023-11-30_08:05:19-fs:shell-wip-batrick-testing-20231130.014408-distro-default-smithi/ * https://tracker.ceph.com/issues/63699 qa: failed cephfs-shell test_reading_conf * https://tracker.ceph.com/issues/63700 qa: test_cd_with_args failure h3. 29 Nov 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231107.042705 * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/62510 (pending RHEL back port) snaptest-git-ceph.sh failure with fs/thrash * https://tracker.ceph.com/issues/62810 Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again h3. 14 Nov 2023 (Milind) https://pulpito.ceph.com/mchangir-2023-11-13_10:27:15-fs-wip-mchangir-testing-20231110.052303-testing-default-smithi/ * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/63521 qa: Test failure: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/62580 testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/63522 No module named 'tasks.ceph_fuse' No module named 'tasks.kclient' No module named 'tasks.cephfs.fuse_mount' No module named 'tasks.ceph' * https://tracker.ceph.com/issues/63523 Command failed - qa/workunits/fs/misc/general_vxattrs.sh h3. 14 Nov 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231106.073650 (nvm the fs:upgrade test failure - the PR is excluded from merge) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/62580 testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/63519 ceph-fuse: reef ceph-fuse crashes with main branch ceph-mds * https://tracker.ceph.com/issues/57087 qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure * https://tracker.ceph.com/issues/58945 qa: xfstests-dev's generic test suite has 20 failures with fuse client h3. 7 Nov 2023 fs: https://pulpito.ceph.com/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/ re-run: https://pulpito.ceph.com/rishabh-2023-11-05_14:10:09-fs-rishabh-2023nov3-testing-default-smithi/ smoke: https://pulpito.ceph.com/rishabh-2023-11-08_08:39:05-smoke-rishabh-2023nov3-testing-default-smithi/ * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63473 fsstress.sh failed with errno 124 h3. 3 Nov 2023 https://pulpito.ceph.com/rishabh-2023-10-27_06:26:52-fs-rishabh-2023oct26-testing-default-smithi/ * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/57656 dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59531 "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" h3. 24 October 2023 https://pulpito.ceph.com/?branch=wip-batrick-testing-20231024.144545 Two failures: https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/ https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/ probably related to https://github.com/ceph/ceph/pull/53255. Killing the mount as part of the test did not complete. Will research more. * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/59531 "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/63411 qa: flush journal may cause timeouts of `scrub status` * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/63141 test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails h3. 18 Oct 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/59531 "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/62658 error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds * https://tracker.ceph.com/issues/62580 testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/62036 src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty()) * https://tracker.ceph.com/issues/58945 qa: xfstests-dev's generic test suite has 20 failures with fuse client * https://tracker.ceph.com/issues/62847 mds: blogbench requests stuck (5mds+scrub+snaps-flush) h3. 13 Oct 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/62936 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/47292 cephfs-shell: test_df_for_valid_file failure * https://tracker.ceph.com/issues/63141 qa/cephfs: test_idem_unaffected_root_squash fails * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout * https://tracker.ceph.com/issues/58945 qa: xfstests-dev's generic test suite has 20 failures with fuse client * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS h3. 16 Oct 2023 https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825 Infrastructure issues: * /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log Host lost. One followup fix: * https://pulpito.ceph.com/pdonnell-2023-10-20_00:33:29-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/ Failures: * https://tracker.ceph.com/issues/56694 qa: avoid blocking forever on hung umount * https://tracker.ceph.com/issues/63089 qa: tasks/mirror times out * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/59531 "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/62658 error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/57656 dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/63233 mon|client|mds: valgrind reports possible leaks in the MDS * https://tracker.ceph.com/issues/63278 kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs) h3. 9 Oct 2023 https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/ * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/63141 test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails * https://tracker.ceph.com/issues/62937 logrotate doesn't support parallel execution on same set of logfiles * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55805 error during scrub thrashing reached max tries in 900 secs h3. 26 Sep 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/62873 qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/62682 mon: no mdsmap broadcast after "fs set joinable" is set to true * https://tracker.ceph.com/issues/63089 qa: tasks/mirror times out h3. 22 Sep 2023 https://pulpito.ceph.com/rishabh-2023-09-12_12:12:15-fs-wip-rishabh-2023sep12-b2-testing-default-smithi/ * https://tracker.ceph.com/issues/59348 qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59531 "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/61574 build failure for mdtest project * https://tracker.ceph.com/issues/62702 fsstress.sh: MDS slow requests for the internal 'rename' requests * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/62863 deadlock in ceph-fuse causes teuthology job to hang and fail * https://tracker.ceph.com/issues/62870 test_cluster_info (tasks.cephfs.test_nfs.TestNFS) * https://tracker.ceph.com/issues/62873 test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) h3. 20 Sep 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout * https://tracker.ceph.com/issues/62658 error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds * https://tracker.ceph.com/issues/62915 qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases * https://tracker.ceph.com/issues/59531 quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/62873 qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/62936 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/62937 Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' * https://tracker.ceph.com/issues/62510 snaptest-git-ceph.sh failure with fs/thrash * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout * https://tracker.ceph.com/issues/62126 test failure: suites/blogbench.sh stops running * https://tracker.ceph.com/issues/62682 mon: no mdsmap broadcast after "fs set joinable" is set to true h3. 19 Sep 2023 http://pulpito.front.sepia.ceph.com/mchangir-2023-09-12_05:40:22-fs-wip-mchangir-testing-20230908.140927-testing-default-smithi/ * https://tracker.ceph.com/issues/58220#note-9 workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure * https://tracker.ceph.com/issues/62702 Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/62873 qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/59413 cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/62482 qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) h3. 13 Sep 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/61243 qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed * https://tracker.ceph.com/issues/62567 postgres workunit times out - MDS_SLOW_REQUEST in logs * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout h3. 2023 Sep 12 https://pulpito.ceph.com/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/ A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably: * Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes. Failures: * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/57656 dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/55805 error scrub thrashing reached max tries in 900 secs * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror * https://tracker.ceph.com/issues/62832 common: config_proxy deadlock during shutdown (and possibly other times) * https://tracker.ceph.com/issues/59413 cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/62567 Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58220#note-9 workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/62847 mds: blogbench requests stuck (5mds+scrub+snaps-flush) * https://tracker.ceph.com/issues/62848 qa: fail_fs upgrade scenario hanging * https://tracker.ceph.com/issues/62081 tasks/fscrypt-common does not finish, timesout h3. 11 Sep 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/59531 quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/61243 qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed * https://tracker.ceph.com/issues/62567 postgres workunit times out - MDS_SLOW_REQUEST in logs h3. 6 Sep 2023 Run 2 https://pulpito.ceph.com/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ * https://tracker.ceph.com/issues/51964 test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59348 test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/61892 test_strays.TestStrays.test_snapshot_remove failed * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/62484 qa: ffsb.sh test failure * https://tracker.ceph.com/issues/62567 Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55805 error scrub thrashing reached max tries in 900 secs h3. 6 Sep 2023 https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/ * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/61892 test_snapshot_remove (test_strays.TestStrays) failed * https://tracker.ceph.com/issues/59348 qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/62556 test_acls: xfstests_dev: python2 is missing * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/57656 dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55805 error scrub thrashing reached max tries in 900 secs * https://tracker.ceph.com/issues/62567 Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" * https://tracker.ceph.com/issues/62702 workunit test suites/fsstress.sh on smithi066 with status 124 h3. 5 Sep 2023 https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/ this run has failures but acc to Adam King these are not relevant and should be ignored * https://tracker.ceph.com/issues/61892 test_snapshot_remove (test_strays.TestStrays) failed * https://tracker.ceph.com/issues/59348 test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" * https://tracker.ceph.com/issues/57656 dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/62187 iozone.sh: line 5: iozone: command not found * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55805 error scrub thrashing reached max tries in 900 secs h3. 31 Aug 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/62187 iozone: command not found * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/59531 quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59413 cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" * https://tracker.ceph.com/issues/62653 qa: unimplemented fcntl command: 1036 with fsstress * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/62658 error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds * https://tracker.ceph.com/issues/62188 AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) h3. 25 Aug 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807 * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/62484 qa: ffsb.sh test failure * https://tracker.ceph.com/issues/59531 quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" * https://tracker.ceph.com/issues/62510 snaptest-git-ceph.sh failure with fs/thrash h3. 24 Aug 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/62510 snaptest-git-ceph.sh failure with fs/thrash * https://tracker.ceph.com/issues/62484 qa: ffsb.sh test failure * https://tracker.ceph.com/issues/57087 qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/62187 iozone: command not found * https://tracker.ceph.com/issues/62188 AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) * https://tracker.ceph.com/issues/62567 postgres workunit times out - MDS_SLOW_REQUEST in logs h3. 22 Aug 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61243 test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/62188 AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) * https://tracker.ceph.com/issues/62510 snaptest-git-ceph.sh failure with fs/thrash * https://tracker.ceph.com/issues/62511 src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down) h3. 14 Aug 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601 * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59531 cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/59684 [kclient bug] Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/61243 (NEW) test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) h3. 28 JULY 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049 * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59531 cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://github.com/ceph/ceph/pull/52556 task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4) * https://tracker.ceph.com/issues/62187 iozone: command not found * https://tracker.ceph.com/issues/61399 qa: build failure for ior (the failed instance is when compiling `mdtest`) * https://tracker.ceph.com/issues/62188 AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) h3. 24 Jul 2023 https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/ There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures - https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/ One more extra run to check if blogbench.sh fail every time: https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/ blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing - https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/ * https://tracker.ceph.com/issues/61892 test_snapshot_remove (test_strays.TestStrays) failed * https://tracker.ceph.com/issues/53859 test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/61982 test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/48773 reached max tries: scrub does not complete * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10010 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/61399 ior build failure * https://tracker.ceph.com/issues/57676 error during scrub thrashing: backtrace * https://tracker.ceph.com/issues/38452 'sudo -u postgres -- pgbench -s 500 -i' failed * https://tracker.ceph.com/issues/62126 blogbench.sh failure h3. 18 July 2023 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61182 cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds * https://tracker.ceph.com/issues/61957 test_client_limits.TestClientLimits.test_client_release_bug * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61892 test_strays.TestStrays.test_snapshot_remove failed * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/44565 src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) * https://tracker.ceph.com/issues/62067 ffsb.sh failure "Resource temporarily unavailable" h3. 17 July 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136 * https://tracker.ceph.com/issues/61982 Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61182 cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds * https://tracker.ceph.com/issues/61957 test_client_limits.TestClientLimits.test_client_release_bug * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61892 test_strays.TestStrays.test_snapshot_remove failed * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/62036 src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty()) * https://tracker.ceph.com/issues/61737 coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific' * https://tracker.ceph.com/issues/44565 src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) h3. 13 July 2023 Run 2 https://pulpito.ceph.com/rishabh-2023-07-08_23:33:40-fs-wip-rishabh-2023Jul9-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/ * https://tracker.ceph.com/issues/61957 test_client_limits.TestClientLimits.test_client_release_bug * https://tracker.ceph.com/issues/61982 Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/61399 ior build failure h3. 13 July 2023 https://pulpito.ceph.com/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/ * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61945 LibCephFS.DelegTimeout failure * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/61982 Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) h3. 13 Jul 2023 https://pulpito.ceph.com/rishabh-2023-07-05_22:21:20-fs-wip-rishabh-2023Jul5-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-07-06_19:33:28-fs-wip-rishabh-2023Jul5-testing-default-smithi/ * https://tracker.ceph.com/issues/61957 test_client_limits.TestClientLimits.test_client_release_bug * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/48773 scrub does not complete: reached max tries * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/58742 xfstests-dev: kcephfs: generic * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror h3. 12 July 2023 https://pulpito.ceph.com/rishabh-2023-07-05_18:32:52-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-07-06_19:46:43-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/ * https://tracker.ceph.com/issues/61892 test_strays.TestStrays.test_snapshot_remove failed * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/58742 xfstests-dev: kcephfs: generic * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61182 cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default * https://tracker.ceph.com/issues/48773 reached max tries: scrub does not complete h3. 05 July 2023 https://pulpito.ceph.com/pdonnell-2023-07-05_03:38:33-fs:libcephfs-wip-pdonnell-testing-20230705.003205-distro-default-smithi/ * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" h3. 27 Jun 2023 https://pulpito.ceph.com/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/ * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/61831 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) h3. 22 June 2023 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61394 qa/quincy: cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log * https://tracker.ceph.com/issues/61762 qa: wait_for_clean: failed before timeout expired * https://tracker.ceph.com/issues/61775 cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests) * https://tracker.ceph.com/issues/44565 src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) * https://tracker.ceph.com/issues/61790 cephfs client to mds comms remain silent after reconnect * https://tracker.ceph.com/issues/61791 snaptest-git-ceph.sh test timed out (job dead) h3. 20 June 2023 https://pulpito.ceph.com/vshankar-2023-06-15_04:58:28-fs-wip-vshankar-testing-20230614.124123-testing-default-smithi/ * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/44565 src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) * https://tracker.ceph.com/issues/61737 coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific' h3. 16 June 2023 https://pulpito.ceph.com/rishabh-2023-05-16_10:39:13-fs-wip-rishabh-2023May15-1524-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-05-17_11:09:48-fs-wip-rishabh-2023May15-1524-testing-default-smithi/ https://pulpito.ceph.com/rishabh-2023-05-18_10:01:53-fs-wip-rishabh-2023May15-1524-testing-default-smithi/ (bins were rebuilt with a subset of orig PRs) https://pulpito.ceph.com/rishabh-2023-06-09_10:19:22-fs-wip-rishabh-2023Jun9-1308-testing-default-smithi/ * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59346 fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/61399 libmpich: undefined references to fi_strerror * https://tracker.ceph.com/issues/58945 xfstests-dev: ceph-fuse: generic * https://tracker.ceph.com/issues/58742 xfstests-dev: kcephfs: generic h3. 24 May 2023 https://pulpito.ceph.com/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/ * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59683 Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests * https://tracker.ceph.com/issues/61399 qa: "[Makefile:299: ior] Error 1" * https://tracker.ceph.com/issues/61265 qa: tasks.cephfs.fuse_mount:process failed to terminate after unmount * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/59346 qa/workunits/fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" * https://tracker.ceph.com/issues/61400 valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59344 qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" * https://tracker.ceph.com/issues/61407 mds: abort on CInode::verify_dirfrags * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/61409 qa: _test_stale_caps does not wait for file flush before stat h3. 15 May 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020-6 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59684 [kclient bug] Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/61148 dbench test results in call trace in dmesg [kclient bug] * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds h3. 11 May 2023 https://pulpito.ceph.com/yuriw-2023-05-10_18:21:40-fs-wip-yuri7-testing-2023-05-10-0742-distro-default-smithi/ * https://tracker.ceph.com/issues/59684 [kclient bug] Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55805 error during scrub thrashing reached max tries in 900 secs * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/58220 Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: * https://tracker.ceph.com/issues/58220#note-9 workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure * https://tracker.ceph.com/issues/59342 qa/workunits/kernel_untar_build.sh failed when compiling the Linux source * https://tracker.ceph.com/issues/58949 test_cephfs.test_disk_quota_exceeeded_error - AssertionError: DiskQuotaExceeded not raised by write * https://tracker.ceph.com/issues/61243 (NEW) test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed h3. 11 May 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.054005 (no fsstress job failure [https://tracker.ceph.com/issues/58340] since https://github.com/ceph/ceph/pull/49553 was included in the branch, however, the PR got updated and needs retest). * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/59683 Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests * https://tracker.ceph.com/issues/59684 [kclient bug] Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) h3. 09 May 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230506.143554 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/59350 qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR * https://tracker.ceph.com/issues/59683 Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests * https://tracker.ceph.com/issues/59684 [kclient bug] Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) * https://tracker.ceph.com/issues/59348 qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) h3. 10 Apr 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230330.105356 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure h3. 31 Mar 2023 run: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/ re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-11_05:54:03-fs-wip-rishabh-2023Mar10-1727-testing-default-smithi/ re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-23_08:27:28-fs-wip-rishabh-2023Mar20-2250-testing-default-smithi/ There were many more re-runs for "failed+dead" jobs as well as for individual jobs. half of the PRs from the batch were removed (gradually over subsequent re-runs). * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58220 Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58220#note-9 workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure * https://tracker.ceph.com/issues/56695 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/58564 workuit dbench failed with error code 1 * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/57580 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) * https://tracker.ceph.com/issues/58940 ceph osd hit ceph_abort * https://tracker.ceph.com/issues/55805 error scrub thrashing reached max tries in 900 secs h3. 30 March 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230315.085747 * https://tracker.ceph.com/issues/58938 qa: xfstests-dev's generic test suite has 7 failures with kclient * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds h3. 29 March 2023 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230317.095222 * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/57087 qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/59230 Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) * https://tracker.ceph.com/issues/58938 qa: xfstests-dev's generic test suite has 7 failures with kclient h3. 13 Mar 2023 * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) h3. 09 Mar 2023 https://pulpito.ceph.com/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2023-03-08_15:12:36-fs-wip-vshankar-testing-20230308.112059-testing-default-smithi/ * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/57087 qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure h3. 07 Mar 2023 https://pulpito.ceph.com/vshankar-2023-03-02_09:21:58-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2023-03-07_05:15:12-fs-wip-vshankar-testing-20230307.030510-testing-default-smithi/ * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/58220 Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58934 snaptest-git-ceph.sh failure with ceph-fuse h3. 28 Feb 2023 https://pulpito.ceph.com/vshankar-2023-02-24_02:11:45-fs-wip-vshankar-testing-20230222.025426-testing-default-smithi/ * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) (teuthology infra issues causing testing delays - merging PRs which have tests passing) h3. 25 Jan 2023 https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/ * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/58220 Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/56011 fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 h3. 30 JAN 2023 run: http://pulpito.front.sepia.ceph.com/rishabh-2022-11-28_08:04:11-fs-wip-rishabh-testing-2022Nov24-1818-testing-default-smithi/ re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/ re-run of re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/ * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/55332 Failure in snaptest-git-ceph.sh * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds * https://tracker.ceph.com/issues/58219 Command crashed: 'ceph-dencoder type inode_backtrace_t import - decode dump_json' * "Failed to load ceph-mgr modules: prometheus" in cluster log" http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7134086 Acc to Venky this was fixed in https://github.com/ceph/ceph/commit/cf6089200d96fc56b08ee17a4e31f19823370dc8 * Created https://tracker.ceph.com/issues/58564 workunit test suites/dbench.sh failed error code 1 h3. 15 Dec 2022 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221215.112736 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/58219 * https://tracker.ceph.com/issues/57655 * qa: fs:mixed-clients kernel_untar_build failure Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/58340 mds: fsstress.sh hangs with multimds h3. 08 Dec 2022 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221130.043104 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221209.043803 (lots of transient git.ceph.com failures) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/57655 qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/58219 Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) * https://tracker.ceph.com/issues/58220 Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/58244 Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) h3. 14 Oct 2022 https://pulpito.ceph.com/vshankar-2022-10-12_04:56:59-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-10-14_04:04:57-fs-wip-vshankar-testing-20221014-072608-testing-default-smithi/ * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/57682 client: ERROR: test_reconnect_after_blocklisted * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 h3. 10 Oct 2022 http://pulpito.front.sepia.ceph.com/rishabh-2022-09-30_19:45:21-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/ reruns * fs-thrash, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-04_13:19:47-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/ * fs-verify, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-05_12:25:37-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/ * cephadm failures also passed after many re-runs: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-06_13:50:51-fs-wip-rishabh-testing-30Sep2022-2-testing-default-smithi/ ** needed this PR to be merged in ceph-ci branch - https://github.com/ceph/ceph/pull/47458 known bugs * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke * https://tracker.ceph.com/issues/57299 qa: test_dump_loads fails with JSONDecodeError * https://tracker.ceph.com/issues/57655 [Exist in main as well] qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim h3. 2022 Sep 29 http://pulpito.front.sepia.ceph.com/rishabh-2022-09-14_12:48:43-fs-wip-rishabh-testing-2022Sep9-1708-testing-default-smithi/ * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/36593 Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/56632 Test failure: test_subvolume_snapshot_clone_quota_exceeded * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing h3. 2022 Sep 26 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220923.171109 * https://tracker.ceph.com/issues/55804 qa failure: pjd link tests failed * https://tracker.ceph.com/issues/57676 qa: error during scrub thrashing: rank damage found: {'backtrace'} * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57580 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/57299 qa: test_dump_loads fails with JSONDecodeError * https://tracker.ceph.com/issues/57280 qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/57677 qa: "1 MDSs behind on trimming (MDS_TRIM)" * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/57446 qa: test_subvolume_snapshot_info_if_orphan_clone fails * https://tracker.ceph.com/issues/57655 [Exist in main as well] qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57682 client: ERROR: test_reconnect_after_blocklisted h3. 2022 Sep 22 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220920.234701 * https://tracker.ceph.com/issues/57299 qa: test_dump_loads fails with JSONDecodeError * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57580 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) * https://tracker.ceph.com/issues/57280 qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/51267 CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:... NEW: * https://tracker.ceph.com/issues/57656 [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) * https://tracker.ceph.com/issues/57655 [Exist in main as well] qa: fs:mixed-clients kernel_untar_build failure * https://tracker.ceph.com/issues/57657 mds: scrub locates mismatch between child accounted_rstats and self rstats Segfault probably caused by: https://github.com/ceph/ceph/pull/47795#issuecomment-1255724799 h3. 2022 Sep 16 https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220905-132828 * https://tracker.ceph.com/issues/57446 qa: test_subvolume_snapshot_info_if_orphan_clone fails * https://tracker.ceph.com/issues/57299 qa: test_dump_loads fails with JSONDecodeError * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57280 qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/48203 https://tracker.ceph.com/issues/36593 qa: quota failure qa: quota failure caused by clients stepping on each other * https://tracker.ceph.com/issues/57580 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) h3. 2022 Aug 26 http://pulpito.front.sepia.ceph.com/rishabh-2022-08-22_17:49:59-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/ http://pulpito.front.sepia.ceph.com/rishabh-2022-08-24_11:56:51-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/ * https://tracker.ceph.com/issues/57206 libcephfs/test.sh: ceph_test_libcephfs_reclaim * https://tracker.ceph.com/issues/56632 Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/36593 Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) h3. 2022 Aug 22 https://pulpito.ceph.com/vshankar-2022-08-12_09:34:24-fs-wip-vshankar-testing1-20220812-072441-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-08-18_04:30:42-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/ (drop problematic PR and re-run) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/51278 mds: "FAILED ceph_assert(!segments.empty())" * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57206 ceph_test_libcephfs_reclaim crashes during test * https://tracker.ceph.com/issues/53859 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) h3. 2022 Aug 12 https://pulpito.ceph.com/vshankar-2022-08-10_04:06:00-fs-wip-vshankar-testing-20220805-190751-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-08-11_12:16:58-fs-wip-vshankar-testing-20220811-145809-testing-default-smithi/ (drop problematic PR and re-run) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 h3. 2022 Aug 04 https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220804-123835 (only mgr/volumes, mgr/stats) Unrealted teuthology failure on rhel h3. 2022 Jul 25 http://pulpito.front.sepia.ceph.com/rishabh-2022-07-22_11:34:20-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/ 1st re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_03:51:19-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi 2nd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/ 3rd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/ 4th (final) re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-28_03:59:01-fs-wip-rishabh-testing-2022Jul28-0143-testing-default-smithi/ * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/36593 Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128~ h3. 2022 July 22 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220721.235756 MDS_HEALTH_DUMMY error in log fixed by followup commit. transient selinux ping failure * https://tracker.ceph.com/issues/56694 qa: avoid blocking forever on hung umount * https://tracker.ceph.com/issues/56695 [RHEL stock] pjd test failures * https://tracker.ceph.com/issues/56696 admin keyring disappears during qa run * https://tracker.ceph.com/issues/56697 qa: fs/snaps fails for fuse * https://tracker.ceph.com/issues/50222 osd: 5.2s0 deep-scrub : stat mismatch * https://tracker.ceph.com/issues/56698 client: FAILED ceph_assert(_size == 0) * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" h3. 2022 Jul 15 http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/ re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/ * https://tracker.ceph.com/issues/53859 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/50222 osd: deep-scrub : stat mismatch * https://tracker.ceph.com/issues/56632 Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) * https://tracker.ceph.com/issues/56634 workunit test fs/snaps/snaptest-intodir.sh * https://tracker.ceph.com/issues/56644 Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) h3. 2022 July 05 http://pulpito.front.sepia.ceph.com/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/ On 1st re-run some jobs passed - http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/ On 2nd re-run only few jobs failed - http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/ http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/ * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/55804 Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/ * https://tracker.ceph.com/issues/56445 Command failed on smithi080 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" * https://tracker.ceph.com/issues/51267 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 * https://tracker.ceph.com/issues/50224 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) h3. 2022 July 04 https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/ (rhel runs were borked due to: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/JSZQFUKVLDND4W33PXDGCABPHNSPT6SS/, tests ran with --filter-out=rhel) * https://tracker.ceph.com/issues/56445 Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" * https://tracker.ceph.com/issues/56446 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" h3. 2022 June 20 https://pulpito.ceph.com/vshankar-2022-06-15_04:03:39-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-06-19_08:22:46-fs-wip-vshankar-testing1-20220619-102531-testing-default-smithi/ * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/55804 qa failure: pjd link tests failed * https://tracker.ceph.com/issues/54108 qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" * https://tracker.ceph.com/issues/55332 Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) h3. 2022 June 13 https://pulpito.ceph.com/pdonnell-2022-06-12_05:08:12-fs:workload-wip-pdonnell-testing-20220612.004943-distro-default-smithi/ * https://tracker.ceph.com/issues/56024 cephadm: removes ceph.conf during qa run causing command failure * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/56012 mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay()) h3. 2022 Jun 13 https://pulpito.ceph.com/vshankar-2022-06-07_00:25:50-fs-wip-vshankar-testing-20220606-223254-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-06-10_01:04:46-fs-wip-vshankar-testing-20220609-175550-testing-default-smithi/ * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/55804 qa failure: pjd link tests failed * https://tracker.ceph.com/issues/56003 client: src/include/xlist.h: 81: FAILED ceph_assert(_size == 0) * https://tracker.ceph.com/issues/56011 fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison * https://tracker.ceph.com/issues/56012 mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay()) h3. 2022 Jun 07 https://pulpito.ceph.com/vshankar-2022-06-06_21:25:41-fs-wip-vshankar-testing1-20220606-230129-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-06-07_10:53:31-fs-wip-vshankar-testing1-20220607-104134-testing-default-smithi/ (rerun after dropping a problematic PR) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure h3. 2022 May 12 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220509-125847 https://pulpito.ceph.com/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/ (drop prs + rerun) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/55332 Failure in snaptest-git-ceph.sh * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/55538 Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) * https://tracker.ceph.com/issues/55258 lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs (cropss up again, though very infrequent) h3. 2022 May 04 https://pulpito.ceph.com/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/ (after dropping PRs) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/55332 Failure in snaptest-git-ceph.sh * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) * https://tracker.ceph.com/issues/55516 qa: fs suite tests failing with "json.decoder.JSONDecodeError: Extra data: line 2 column 82 (char 82)" * https://tracker.ceph.com/issues/55537 mds: crash during fs:upgrade test * https://tracker.ceph.com/issues/55538 Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) h3. 2022 Apr 25 https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220420-113951 (owner vshankar) * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/55258 lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs * https://tracker.ceph.com/issues/55377 kclient: mds revoke Fwb caps stuck after the kclient tries writebcak once h3. 2022 Apr 14 https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220411-144044 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/55170 mds: crash during rejoin (CDir::fetch_keys) * https://tracker.ceph.com/issues/55331 pjd failure * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/55332 Failure in snaptest-git-ceph.sh * https://tracker.ceph.com/issues/55258 lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs h3. 2022 Apr 11 https://pulpito.ceph.com/?branch=wip-vshankar-testing-55110-20220408-203242 * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/48680 mds: scrubbing stuck "scrub active (0 inodes in the stack)" * https://tracker.ceph.com/issues/55236 qa: fs/snaps tests fails with "hit max job timeout" * https://tracker.ceph.com/issues/54108 qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" * https://tracker.ceph.com/issues/54971 Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/55258 lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs h3. 2022 Mar 21 https://pulpito.ceph.com/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/ Run didn't go well, lots of failures - debugging by dropping PRs and running against master branch. Only merging unrelated PRs that pass tests. h3. 2022 Mar 08 https://pulpito.ceph.com/vshankar-2022-02-28_04:32:15-fs-wip-vshankar-testing-20220226-211550-testing-default-smithi/ rerun with - (drop) https://github.com/ceph/ceph/pull/44679 - (drop) https://github.com/ceph/ceph/pull/44958 https://pulpito.ceph.com/vshankar-2022-03-06_14:47:51-fs-wip-vshankar-testing-20220304-132102-testing-default-smithi/ * https://tracker.ceph.com/issues/54419 (new) `ceph orch upgrade start` seems to never reach completion * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing h3. 2022 Feb 09 https://pulpito.ceph.com/vshankar-2022-02-05_17:27:49-fs-wip-vshankar-testing-20220201-113815-testing-default-smithi/ rerun with - (drop) https://github.com/ceph/ceph/pull/37938 - (drop) https://github.com/ceph/ceph/pull/44335 - (drop) https://github.com/ceph/ceph/pull/44491 - (drop) https://github.com/ceph/ceph/pull/44501 https://pulpito.ceph.com/vshankar-2022-02-08_14:27:29-fs-wip-vshankar-testing-20220208-181241-testing-default-smithi/ * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/54066 test_subvolume_no_upgrade_v1_sanity fails with `AssertionError: 1000 != 0` * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout h3. 2022 Feb 01 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220127.171526 * https://tracker.ceph.com/issues/54107 kclient: hang during umount * https://tracker.ceph.com/issues/54106 kclient: hang during workunit cleanup * https://tracker.ceph.com/issues/54108 qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/52438 qa: ffsb timeout h3. 2022 Jan 13 https://pulpito.ceph.com/vshankar-2022-01-06_13:18:41-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/ rerun with: - (add) https://github.com/ceph/ceph/pull/44570 - (drop) https://github.com/ceph/ceph/pull/43184 https://pulpito.ceph.com/vshankar-2022-01-13_04:42:40-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/ * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/53859 qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) h3. 2022 Jan 03 https://pulpito.ceph.com/vshankar-2021-12-22_07:37:44-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2022-01-03_12:27:45-fs-wip-vshankar-testing-20220103-142738-testing-default-smithi/ (rerun) * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/51267 CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:... * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/51278 mds: "FAILED ceph_assert(!segments.empty())" * https://tracker.ceph.com/issues/52279 cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter h3. 2021 Dec 22 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211222.014316 * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52279 cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure * https://tracker.ceph.com/issues/48773 qa: scrub does not complete h3. 2021 Nov 30 https://pulpito.ceph.com/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/ https://pulpito.ceph.com/vshankar-2021-11-30_06:23:32-fs-wip-vshankar-testing-20211124-094330-distro-default-smithi/ (rerun w/ QA fixes) * https://tracker.ceph.com/issues/53436 mds, mon: mds beacon messages get dropped? (mds never reaches up:active state) * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/48812 qa: test_scrub_pause_and_resume_with_abort failure * https://tracker.ceph.com/issues/51076 "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend. * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") h3. 2021 November 9 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211109.180315 * https://tracker.ceph.com/issues/53214 qa: "dd: error reading '/sys/kernel/debug/ceph/2a934501-6731-4052-a836-f42229a869be.client4874/metrics': Is a directory" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/53216 qa: "RuntimeError: value of attributes should be either str or None. client_id" * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") h3. 2021 November 03 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211103.023355 * https://tracker.ceph.com/issues/51964 qa: test_cephfs_mirror_restart_sync_on_blocklist failure * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/52436 fs/ceph: "corrupt mdsmap" * https://tracker.ceph.com/issues/53074 pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active * https://tracker.ceph.com/issues/53150 pybind/mgr/cephadm/upgrade: tolerate MDS failures during upgrade straddling v16.2.5 * https://tracker.ceph.com/issues/53155 MDSMonitor: assertion during upgrade to v16.2.5+ h3. 2021 October 26 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211025.000447 * https://tracker.ceph.com/issues/53074 pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active * https://tracker.ceph.com/issues/52997 testing: hang ing umount * https://tracker.ceph.com/issues/50824 qa: snaptest-git-ceph bus error * https://tracker.ceph.com/issues/52436 fs/ceph: "corrupt mdsmap" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/53082 ceph-fuse: segmenetation fault in Client::handle_mds_map * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") h3. 2021 October 19 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211019.013028 * https://tracker.ceph.com/issues/52995 qa: test_standby_count_wanted failure * https://tracker.ceph.com/issues/52948 osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up" * https://tracker.ceph.com/issues/52996 qa: test_perf_counters via test_openfiletable * https://tracker.ceph.com/issues/48772 qa: pjd: not ok 9, 44, 80 * https://tracker.ceph.com/issues/52997 testing: hang ing umount * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/48773 qa: scrub does not complete h3. 2021 October 12 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211012.192211 Some failures caused by teuthology bug: https://tracker.ceph.com/issues/52944 New test caused failure: https://github.com/ceph/ceph/pull/43297#discussion_r729883167 * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/52948 osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure * https://tracker.ceph.com/issues/52949 RuntimeError: The following counters failed to be set on mds daemons: {'mds.dir_split'} h3. 2021 October 02 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211002.163337 Some failures caused by cephadm upgrade test. Fixed in follow-up qa commit. test_simple failures caused by PR in this set. A few reruns because of QA infra noise. * https://tracker.ceph.com/issues/52822 qa: failed pacific install on fs:upgrade * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete h3. 2021 September 20 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210917.174826 * https://tracker.ceph.com/issues/52677 qa: test_simple failure * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout h3. 2021 September 10 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210910.181451 * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") * https://tracker.ceph.com/issues/52624 qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" * https://tracker.ceph.com/issues/52625 qa: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) * https://tracker.ceph.com/issues/52439 qa: acls does not compile on centos stream * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/52626 mds: ScrubStack.cc: 831: FAILED ceph_assert(diri) * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) h3. 2021 August 27 Several jobs died because of device failures. https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210827.024746 * https://tracker.ceph.com/issues/52430 mds: fast async create client mount breaks racy test * https://tracker.ceph.com/issues/52436 fs/ceph: "corrupt mdsmap" * https://tracker.ceph.com/issues/52437 mds: InoTable::replay_release_ids abort via test_inotable_sync * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/52438 qa: ffsb timeout * https://tracker.ceph.com/issues/52439 qa: acls does not compile on centos stream h3. 2021 July 30 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210729.214022 * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51975 pybind/mgr/stats: KeyError h3. 2021 July 28 https://pulpito.ceph.com/pdonnell-2021-07-28_00:39:45-fs-wip-pdonnell-testing-20210727.213757-distro-basic-smithi/ with qa fix: https://pulpito.ceph.com/pdonnell-2021-07-28_16:20:28-fs-wip-pdonnell-testing-20210728.141004-distro-basic-smithi/ * https://tracker.ceph.com/issues/51905 qa: "error reading sessionmap 'mds1_sessionmap'" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") * https://tracker.ceph.com/issues/51267 CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:... * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) h3. 2021 July 16 https://pulpito.ceph.com/pdonnell-2021-07-16_05:50:11-fs-wip-pdonnell-testing-20210716.022804-distro-basic-smithi/ * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/48772 qa: pjd: not ok 9, 44, 80 * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) * https://tracker.ceph.com/issues/50824 qa: snaptest-git-ceph bus error h3. 2021 July 04 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210703.052904 * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/39150 mon: "FAILED ceph_assert(session_map.sessions.empty())" when out of quorum * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") h3. 2021 July 01 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210701.192056 * https://tracker.ceph.com/issues/51197 qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details * https://tracker.ceph.com/issues/50866 osd: stat mismatch on objects * https://tracker.ceph.com/issues/48773 qa: scrub does not complete h3. 2021 June 26 https://pulpito.ceph.com/pdonnell-2021-06-26_00:57:00-fs-wip-pdonnell-testing-20210625.225421-distro-basic-smithi/ * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/51410 kclient: fails to finish reconnect during MDS thrashing (testing branch) * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/48772 qa: pjd: not ok 9, 44, 80 h3. 2021 June 21 https://pulpito.ceph.com/pdonnell-2021-06-22_00:27:21-fs-wip-pdonnell-testing-20210621.231646-distro-basic-smithi/ One failure caused by PR: https://github.com/ceph/ceph/pull/41935#issuecomment-866472599 * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/50495 libcephfs: shutdown race fails with status 141 * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/50824 qa: snaptest-git-ceph bus error * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" h3. 2021 June 16 https://pulpito.ceph.com/pdonnell-2021-06-16_21:26:55-fs-wip-pdonnell-testing-20210616.191804-distro-basic-smithi/ MDS abort class of failures caused by PR: https://github.com/ceph/ceph/pull/41667 * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/43216 MDSMonitor: removes MDS coming out of quorum election * https://tracker.ceph.com/issues/51278 mds: "FAILED ceph_assert(!segments.empty())" * https://tracker.ceph.com/issues/51279 kclient hangs on umount (testing branch) * https://tracker.ceph.com/issues/51280 mds: "FAILED ceph_assert(r == 0 || r == -2)" * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/51281 qa: snaptest-snap-rm-cmp.sh: "echo 'FAIL: bad match, /tmp/a 4637e766853d1ad16a7b17079e2c6f03 != real c3883760b18d50e8d78819c54d579b00'" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51076 "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend. * https://tracker.ceph.com/issues/51228 qa: rmdir: failed to remove 'a/.snap/*': No such file or directory * https://tracker.ceph.com/issues/51282 pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings h3. 2021 June 14 https://pulpito.ceph.com/pdonnell-2021-06-14_20:53:05-fs-wip-pdonnell-testing-20210614.173325-distro-basic-smithi/ Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific. * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/51228 qa: rmdir: failed to remove 'a/.snap/*': No such file or directory * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/51182 pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' * https://tracker.ceph.com/issues/51229 qa: test_multi_snap_schedule list difference failure * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing h3. 2021 June 13 https://pulpito.ceph.com/pdonnell-2021-06-12_02:45:35-fs-wip-pdonnell-testing-20210612.002809-distro-basic-smithi/ Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific. * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51182 pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/51197 qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed h3. 2021 June 11 https://pulpito.ceph.com/pdonnell-2021-06-11_18:02:10-fs-wip-pdonnell-testing-20210611.162716-distro-basic-smithi/ Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific. * https://tracker.ceph.com/issues/51169 qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing * https://tracker.ceph.com/issues/43216 MDSMonitor: removes MDS coming out of quorum election * https://tracker.ceph.com/issues/51182 pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/51183 qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' * https://tracker.ceph.com/issues/51184 qa: fs:bugs does not specify distro h3. 2021 June 03 https://pulpito.ceph.com/pdonnell-2021-06-03_03:40:33-fs-wip-pdonnell-testing-20210603.020013-distro-basic-smithi/ * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/49845#note-2 (regression) qa: failed umount in test_volumes * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/43216 MDSMonitor: removes MDS coming out of quorum election h3. 2021 May 18 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.214114 Regression in testing kernel caused some failures. Ilya fixed those and rerun looked better. Some odd new noise in the rerun relating to packaging and "No module named 'tasks.ceph'". * https://tracker.ceph.com/issues/50824 qa: snaptest-git-ceph bus error * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/49845#note-2 (regression) qa: failed umount in test_volumes * https://tracker.ceph.com/issues/48203 (stock kernel update required) qa: quota failure h3. 2021 May 18 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.025642 * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/45591 mgr: FAILED ceph_assert(daemon != nullptr) * https://tracker.ceph.com/issues/50866 osd: stat mismatch on objects * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" * https://tracker.ceph.com/issues/50867 qa: fs:mirror: reduced data availability * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50868 qa: "kern.log.gz already exists; not overwritten" * https://tracker.ceph.com/issues/50870 qa: test_full: "rm: cannot remove 'large_file_a': Permission denied" h3. 2021 May 11 https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210511.232042 * one class of failures caused by PR * https://tracker.ceph.com/issues/48812 qa: test_scrub_pause_and_resume_with_abort failure * https://tracker.ceph.com/issues/50390 mds: monclient: wait_auth_rotating timed out after 30 * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/50825 qa: snaptest-git-ceph hang during mon thrashing v2 * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50823 qa: RuntimeError: timeout waiting for cluster to stabilize h3. 2021 May 14 https://pulpito.ceph.com/pdonnell-2021-05-14_21:45:42-fs-master-distro-basic-smithi/ * https://tracker.ceph.com/issues/48812 qa: test_scrub_pause_and_resume_with_abort failure * https://tracker.ceph.com/issues/50821 qa: untar_snap_rm failure during mds thrashing * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/50822 qa: testing kernel patch for client metrics causes mds abort * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50823 qa: RuntimeError: timeout waiting for cluster to stabilize * https://tracker.ceph.com/issues/50824 qa: snaptest-git-ceph bus error * https://tracker.ceph.com/issues/50825 qa: snaptest-git-ceph hang during mon thrashing v2 * https://tracker.ceph.com/issues/50826 kceph: stock RHEL kernel hangs on snaptests with mon|osd thrashers h3. 2021 May 01 https://pulpito.ceph.com/pdonnell-2021-05-01_09:07:09-fs-wip-pdonnell-testing-20210501.040415-distro-basic-smithi/ * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/50281 qa: untar_snap_rm timeout * https://tracker.ceph.com/issues/48203 (stock kernel update required) qa: quota failure * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50390 mds: monclient: wait_auth_rotating timed out after 30 * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/50622 (regression) msg: active_connections regression * https://tracker.ceph.com/issues/45591 mgr: FAILED ceph_assert(daemon != nullptr) * https://tracker.ceph.com/issues/50221 qa: snaptest-git-ceph failure in git diff * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" h3. 2021 Apr 15 https://pulpito.ceph.com/pdonnell-2021-04-15_01:35:57-fs-wip-pdonnell-testing-20210414.230315-distro-basic-smithi/ * https://tracker.ceph.com/issues/50281 qa: untar_snap_rm timeout * https://tracker.ceph.com/issues/50220 qa: dbench workload timeout * https://tracker.ceph.com/issues/50246 mds: failure replaying journal (EMetaBlob) * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" * https://tracker.ceph.com/issues/50222 osd: 5.2s0 deep-scrub : stat mismatch * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/49845 qa: failed umount in test_volumes * https://tracker.ceph.com/issues/37808 osd: osdmap cache weak_refs assert during shutdown * https://tracker.ceph.com/issues/50387 client: fs/snaps failure * https://tracker.ceph.com/issues/50389 mds: "cluster [ERR] Error recovering journal 0x203: (2) No such file or directory" in cluster log" * https://tracker.ceph.com/issues/50216 qa: "ls: cannot access 'lost+found': No such file or directory" * https://tracker.ceph.com/issues/50390 mds: monclient: wait_auth_rotating timed out after 30 h3. 2021 Apr 08 https://pulpito.ceph.com/pdonnell-2021-04-08_22:42:24-fs-wip-pdonnell-testing-20210408.192301-distro-basic-smithi/ * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/50279 qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c" * https://tracker.ceph.com/issues/50246 mds: failure replaying journal (EMetaBlob) * https://tracker.ceph.com/issues/48365 qa: ffsb build failure on CentOS 8.2 * https://tracker.ceph.com/issues/50216 qa: "ls: cannot access 'lost+found': No such file or directory" * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50280 cephadm: RuntimeError: uid/gid not found * https://tracker.ceph.com/issues/50281 qa: untar_snap_rm timeout h3. 2021 Apr 08 https://pulpito.ceph.com/pdonnell-2021-04-08_04:31:36-fs-wip-pdonnell-testing-20210408.024225-distro-basic-smithi/ https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210408.142238 (with logic inversion / QA fix) * https://tracker.ceph.com/issues/50246 mds: failure replaying journal (EMetaBlob) * https://tracker.ceph.com/issues/50250 mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" h3. 2021 Apr 07 https://pulpito.ceph.com/pdonnell-2021-04-07_02:12:41-fs-wip-pdonnell-testing-20210406.213012-distro-basic-smithi/ * https://tracker.ceph.com/issues/50215 qa: "log [ERR] : error reading sessionmap 'mds2_sessionmap'" * https://tracker.ceph.com/issues/49466 qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" * https://tracker.ceph.com/issues/50216 qa: "ls: cannot access 'lost+found': No such file or directory" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/49845 qa: failed umount in test_volumes * https://tracker.ceph.com/issues/50220 qa: dbench workload timeout * https://tracker.ceph.com/issues/50221 qa: snaptest-git-ceph failure in git diff * https://tracker.ceph.com/issues/50222 osd: 5.2s0 deep-scrub : stat mismatch * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/50224 qa: test_mirroring_init_failure_with_recovery failure h3. 2021 Apr 01 https://pulpito.ceph.com/pdonnell-2021-04-01_00:45:34-fs-wip-pdonnell-testing-20210331.222326-distro-basic-smithi/ * https://tracker.ceph.com/issues/48772 qa: pjd: not ok 9, 44, 80 * https://tracker.ceph.com/issues/50177 osd: "stalled aio... buggy kernel or bad device?" * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing * https://tracker.ceph.com/issues/49845 qa: failed umount in test_volumes * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/48805 mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/50178 qa: "TypeError: run() got an unexpected keyword argument 'shell'" * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed h3. 2021 Mar 24 https://pulpito.ceph.com/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/ * https://tracker.ceph.com/issues/49500 qa: "Assertion `cb_done' failed." * https://tracker.ceph.com/issues/50019 qa: mount failure with cephadm "probably no MDS server is up?" * https://tracker.ceph.com/issues/50020 qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/48805 mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/48772 qa: pjd: not ok 9, 44, 80 * https://tracker.ceph.com/issues/50021 qa: snaptest-git-ceph failure during mon thrashing * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing * https://tracker.ceph.com/issues/50016 qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" * https://tracker.ceph.com/issues/49466 qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" h3. 2021 Mar 18 https://pulpito.ceph.com/pdonnell-2021-03-18_13:46:31-fs-wip-pdonnell-testing-20210318.024145-distro-basic-smithi/ * https://tracker.ceph.com/issues/49466 qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/48805 mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/49845 qa: failed umount in test_volumes * https://tracker.ceph.com/issues/49605 mgr: drops command on the floor * https://tracker.ceph.com/issues/48203 (stock kernel update required) qa: quota failure * https://tracker.ceph.com/issues/49928 client: items pinned in cache preventing unmount x2 h3. 2021 Mar 15 https://pulpito.ceph.com/pdonnell-2021-03-15_22:16:56-fs-wip-pdonnell-testing-20210315.182203-distro-basic-smithi/ * https://tracker.ceph.com/issues/49842 qa: stuck pkg install * https://tracker.ceph.com/issues/49466 qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" * https://tracker.ceph.com/issues/49822 test: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) failure * https://tracker.ceph.com/issues/49240 terminate called after throwing an instance of 'std::bad_alloc' * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/49500 qa: "Assertion `cb_done' failed." * https://tracker.ceph.com/issues/49843 qa: fs/snaps/snaptest-upchildrealms.sh failure * https://tracker.ceph.com/issues/49845 qa: failed umount in test_volumes * https://tracker.ceph.com/issues/48805 mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/49605 mgr: drops command on the floor and failure caused by PR: https://github.com/ceph/ceph/pull/39969 h3. 2021 Mar 09 https://pulpito.ceph.com/pdonnell-2021-03-09_03:27:39-fs-wip-pdonnell-testing-20210308.214827-distro-basic-smithi/ * https://tracker.ceph.com/issues/49500 qa: "Assertion `cb_done' failed." * https://tracker.ceph.com/issues/48805 mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" * https://tracker.ceph.com/issues/48773 qa: scrub does not complete * https://tracker.ceph.com/issues/45434 qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed * https://tracker.ceph.com/issues/49240 terminate called after throwing an instance of 'std::bad_alloc' * https://tracker.ceph.com/issues/49466 qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" * https://tracker.ceph.com/issues/49684 qa: fs:cephadm mount does not wait for mds to be created * https://tracker.ceph.com/issues/48771 qa: iogen: workload fails to cause balancing