Ceph » CephFS

h1. <code>main</code> branch

h3. 2024-03-28

https://tracker.ceph.com/issues/65213

* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676

* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572

* "PG_DEGRADED warnings during cluster creation via cephadm: Health check failed: Degraded data":https://tracker.ceph.com/issues/65018

h3. 2024-03-25

https://pulpito.ceph.com/mchangir-2024-03-22_09:46:06-fs:upgrade-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/

* https://tracker.ceph.com/issues/64502

  fusermount -u fails with: teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds

https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/

* https://tracker.ceph.com/issues/62245

  libcephfs/test.sh failed - https://tracker.ceph.com/issues/62245#note-3

h3. 2024-03-20

https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742

https://github.com/batrick/ceph/commit/360516069d9393362c4cc6eb9371680fe16d66ab

Ubuntu jobs filtered out because builds were skipped by jenkins/shaman.

This run has a lot more failures because https://github.com/ceph/ceph/pull/55455 fixed log WRN/ERR checks.

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/65018

    PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)"

* https://tracker.ceph.com/issues/64707 (new issue)

    suites/fsstress.sh hangs on one client - test times out

* https://tracker.ceph.com/issues/64988

    qa: fs:workloads mgr client evicted indicated by "cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds"

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/64972

    qa: "ceph tell 4.3a deep-scrub" command not found

* https://tracker.ceph.com/issues/54108

    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"

* https://tracker.ceph.com/issues/65019

    qa/suites/fs/top: [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 

* https://tracker.ceph.com/issues/65020

    qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log

* https://tracker.ceph.com/issues/65021

    qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64711

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/50821

    qa: untar_snap_rm failure during mds thrashing

* https://tracker.ceph.com/issues/65022

    qa: test_max_items_per_obj open procs not fully cleaned up

h3.  14th March 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240307.013758

(pjd.sh failures are related to a bug in the testing kernel. See - https://tracker.ceph.com/issues/64679#note-4)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64502

    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

h3. 5th March 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240304.042522

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64502

    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/64707 (new issue)

    suites/fsstress.sh hangs on one client - test times out

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/64711

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/64729 (new issue)

    mon.a (mon.0) 1281 : cluster 3 [WRN] MDS_SLOW_METADATA_IO: 3 MDSs report slow metadata IOs" in cluster log

* https://tracker.ceph.com/issues/64730

    fs/misc/multiple_rsync.sh workunit times out

h3. 26th Feb 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240216.060239

(This run is a bit messy due to

  a) OCI runtime issues in the testing kernel with centos9

  b) SELinux denials related failures

  c) Unrelated MON_DOWN warnings)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64172

    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

h3. 20th Feb 2024

https://github.com/ceph/ceph/pull/55601

https://github.com/ceph/ceph/pull/55659

https://pulpito.ceph.com/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/

* https://tracker.ceph.com/issues/64502

    client: quincy ceph-fuse fails to unmount after upgrade to main

This run has numerous problems. #55601 introduces testing for the upgrade sequence from </code>reef/{v18.2.0,v18.2.1,reef}</code> as well as an extra dimension for the ceph-fuse client. The main "big" issue is i64502: the ceph-fuse client is not being unmounted when <code>fusermount -u</code> is called. Instead, the client begins to unmount only after daemons are shut down during test cleanup.

h3. 19th Feb 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240217.015652

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64482

    ceph: stderr Error: OCI runtime error: crun: bpf create ``: Function not implemented

h3. 29 Jan 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240119.075157-1

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/64172

    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/64209

    snaptest-multiple-capsnaps.sh fails with "got remote process result: 1"

h3. 17th Jan 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240103.072409-1

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63259

    mds: failed to store backtrace and force file system read-only

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

h3. 16 Jan 2024

https://pulpito.ceph.com/rishabh-2023-12-11_15:37:57-fs-rishabh-2023dec11-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-12-17_11:19:43-fs-rishabh-2023dec11-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2024-01-04_18:43:16-fs-rishabh-2024jan4-testing-default-smithi

* https://tracker.ceph.com/issues/63764

  Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63141

  qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable" 

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/54462 

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63949

  valgrind leak in MDS

* https://tracker.ceph.com/issues/64041

  qa/cephfs: fs/upgrade/nofs suite attempts to jump more than 2 releases

* fsstress failure in last run was due a kernel MM layer failure, unrelated to CephFS

* from last run, job #7507400 failed due to MGR; FS wasn't degraded, so it's unrelated to CephFS

h3. 06 Dec 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818-x (rerun w/ squid kickoff changes)

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

* https://tracker.ceph.com/issues/63806

    ffsb.sh workunit failure (MDS: std::out_of_range, damaged)

h3. 30 Nov 2023

https://pulpito.ceph.com/pdonnell-2023-11-30_08:05:19-fs:shell-wip-batrick-testing-20231130.014408-distro-default-smithi/

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

h3. 29 Nov 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231107.042705

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62510 (pending RHEL back port)
    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62810

    Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again

h3. 14 Nov 2023

(Milind)

https://pulpito.ceph.com/mchangir-2023-11-13_10:27:15-fs-wip-mchangir-testing-20231110.052303-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63521

  qa: Test failure: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63522

    No module named 'tasks.ceph_fuse'

    No module named 'tasks.kclient'

    No module named 'tasks.cephfs.fuse_mount'

    No module named 'tasks.ceph'

* https://tracker.ceph.com/issues/63523

    Command failed - qa/workunits/fs/misc/general_vxattrs.sh

h3. 14 Nov 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231106.073650

(nvm the fs:upgrade test failure - the PR is excluded from merge)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63519

    ceph-fuse: reef ceph-fuse crashes with main branch ceph-mds

* https://tracker.ceph.com/issues/57087

    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

h3. 7 Nov 2023

fs: https://pulpito.ceph.com/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/

re-run: https://pulpito.ceph.com/rishabh-2023-11-05_14:10:09-fs-rishabh-2023nov3-testing-default-smithi/

smoke: https://pulpito.ceph.com/rishabh-2023-11-08_08:39:05-smoke-rishabh-2023nov3-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63473

  fsstress.sh failed with errno 124

h3. 3 Nov 2023

https://pulpito.ceph.com/rishabh-2023-10-27_06:26:52-fs-rishabh-2023oct26-testing-default-smithi/

* https://tracker.ceph.com/issues/63141

  qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/52624

  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

h3. 24 October 2023

https://pulpito.ceph.com/?branch=wip-batrick-testing-20231024.144545

Two failures:

https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/

https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/

probably related to https://github.com/ceph/ceph/pull/53255. Killing the mount as part of the test did not complete. Will research more.

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/59531

    "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS 

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/63411

    qa: flush journal may cause timeouts of `scrub status`

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63141

    test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails

h3. 18 Oct 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/62658

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62036

    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

* https://tracker.ceph.com/issues/62847

    mds: blogbench requests stuck (5mds+scrub+snaps-flush)

h3. 13 Oct 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62936

    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/47292

    cephfs-shell: test_df_for_valid_file failure

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

h3. 16 Oct 2023

https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825

Infrastructure issues:

* /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log

    Host lost.

One followup fix:

* https://pulpito.ceph.com/pdonnell-2023-10-20_00:33:29-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/

Failures:

* https://tracker.ceph.com/issues/56694

    qa: avoid blocking forever on hung umount

* https://tracker.ceph.com/issues/63089

    qa: tasks/mirror times out

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62658 

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/61243

  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63278

  kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs)

h3. 9 Oct 2023

https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/63141

  test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails

* https://tracker.ceph.com/issues/62937

  logrotate doesn't support parallel execution on same set of logfiles

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error during scrub thrashing reached max tries in 900 secs

h3. 26 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62873

    qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62682

    mon: no mdsmap broadcast after "fs set joinable" is set to true

* https://tracker.ceph.com/issues/63089

    qa: tasks/mirror times out

h3. 22 Sep 2023

https://pulpito.ceph.com/rishabh-2023-09-12_12:12:15-fs-wip-rishabh-2023sep12-b2-testing-default-smithi/

* https://tracker.ceph.com/issues/59348

  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/61574

  build failure for mdtest project

* https://tracker.ceph.com/issues/62702

  fsstress.sh: MDS slow requests for the internal 'rename' requests

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62863 

  deadlock in ceph-fuse causes teuthology job to hang and fail

* https://tracker.ceph.com/issues/62870

  test_cluster_info (tasks.cephfs.test_nfs.TestNFS)

* https://tracker.ceph.com/issues/62873

  test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

h3. 20 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/62658 

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62915

    qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/62873

  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/62936

    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/62937

    Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/62126

    test failure: suites/blogbench.sh stops running

* https://tracker.ceph.com/issues/62682

    mon: no mdsmap broadcast after "fs set joinable" is set to true

h3. 19 Sep 2023

http://pulpito.front.sepia.ceph.com/mchangir-2023-09-12_05:40:22-fs-wip-mchangir-testing-20230908.140927-testing-default-smithi/

* https://tracker.ceph.com/issues/58220#note-9

  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure

* https://tracker.ceph.com/issues/62702

  Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/52624

  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61243

  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/62873

  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/59413

  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/62482

  qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)

h3. 13 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909

* https://tracker.ceph.com/issues/52624

      qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

    qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed

* https://tracker.ceph.com/issues/62567

    postgres workunit times out - MDS_SLOW_REQUEST in logs

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

h3. 2023 Sep 12

https://pulpito.ceph.com/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/

A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably:

* Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes.

Failures:

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61399

  libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/62832

  common: config_proxy deadlock during shutdown (and possibly other times)

* https://tracker.ceph.com/issues/59413

  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62567

  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/58220#note-9

  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/62847

    mds: blogbench requests stuck (5mds+scrub+snaps-flush)

* https://tracker.ceph.com/issues/62848

    qa: fail_fs upgrade scenario hanging

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

h3. 11 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

  qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed

* https://tracker.ceph.com/issues/62567

  postgres workunit times out - MDS_SLOW_REQUEST in logs

h3. 6 Sep 2023 Run 2

https://pulpito.ceph.com/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ 

* https://tracker.ceph.com/issues/51964

  test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59348

  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/61892

  test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/62484

  qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/62567

  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

h3. 6 Sep 2023

https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/59348

  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/54462

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/62556

  test_acls: xfstests_dev: python2 is missing

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

* https://tracker.ceph.com/issues/62567

  Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/62702

  workunit test suites/fsstress.sh on smithi066 with status 124

h3. 5 Sep 2023

https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/

orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/

  this run has failures but acc to Adam King these are not relevant and should be ignored

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/59348

  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/54462

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57656 

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/50223

  client.xxxx isn't responding to mclientcaps(revoke)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62187

  iozone.sh: line 5: iozone: command not found

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

h3. 31 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59413

    cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/62653

    qa: unimplemented fcntl command: 1036 with fsstress

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/62658

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

h3. 25 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62484

    qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

h3. 24 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62484

    qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/57087

    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

* https://tracker.ceph.com/issues/62567

    postgres workunit times out - MDS_SLOW_REQUEST in logs

h3. 22 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62511

    src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down)

h3. 14 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59531

    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/59684 [kclient bug]

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243 (NEW)

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

h3. 28 JULY 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59531

    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://github.com/ceph/ceph/pull/52556

    task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4)

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

h3. 24 Jul 2023

https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/

There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures -

https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/

One more extra run to check if blogbench.sh fail every time:

https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/

blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing -

https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/53859

  test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/61982

  test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

* https://tracker.ceph.com/issues/52438

  qa: ffsb timeout

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/48773

  reached max tries: scrub does not complete

* https://tracker.ceph.com/issues/58340

  mds: fsstress.sh hangs with multimds

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/57206

  libcephfs/test.sh: ceph_test_libcephfs_reclaim

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  error during scrub thrashing: backtrace

* https://tracker.ceph.com/issues/38452

  'sudo -u postgres -- pgbench -s 500 -i' failed

* https://tracker.ceph.com/issues/62126

  blogbench.sh failure

h3. 18 July 2023

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/54460

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61182

    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds

* https://tracker.ceph.com/issues/61957

    test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61892

    test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/44565

    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

h3. 17 July 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136

* https://tracker.ceph.com/issues/61982

    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61182

    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds

* https://tracker.ceph.com/issues/61957

    test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61892

    test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/62036

    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())

* https://tracker.ceph.com/issues/61737

    coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'

* https://tracker.ceph.com/issues/44565

    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())

h3. 13 July 2023 Run 2

https://pulpito.ceph.com/rishabh-2023-07-08_23:33:40-fs-wip-rishabh-2023Jul9-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/

* https://tracker.ceph.com/issues/61957

  test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/61982

  Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

  ior build failure

h3. 13 July 2023

https://pulpito.ceph.com/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/

* https://tracker.ceph.com/issues/54460

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61945

    LibCephFS.DelegTimeout failure

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/61982

    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

h3. 13 Jul 2023

https://pulpito.ceph.com/rishabh-2023-07-05_22:21:20-fs-wip-rishabh-2023Jul5-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-07-06_19:33:28-fs-wip-rishabh-2023Jul5-testing-default-smithi/

* https://tracker.ceph.com/issues/61957

  test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/48773

  scrub does not complete: reached max tries

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/52438

  qa: ffsb timeout

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/58742

  xfstests-dev: kcephfs: generic

* https://tracker.ceph.com/issues/61399

  libmpich: undefined references to fi_strerror

h3. 12 July 2023

https://pulpito.ceph.com/rishabh-2023-07-05_18:32:52-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-07-06_19:46:43-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/

* https://tracker.ceph.com/issues/61892

  test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/58742

  xfstests-dev: kcephfs: generic

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/52438

  qa: ffsb timeout

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61182

  cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/48773

  reached max tries: scrub does not complete

h3. 05 July 2023

https://pulpito.ceph.com/pdonnell-2023-07-05_03:38:33-fs:libcephfs-wip-pdonnell-testing-20230705.003205-distro-default-smithi/

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

h3. 27 Jun 2023

https://pulpito.ceph.com/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/61399

  libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/50223

  client.xxxx isn't responding to mclientcaps(revoke)

* https://tracker.ceph.com/issues/61831

  Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

h3. 22 June 2023

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61394

    qa/quincy: cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log

* https://tracker.ceph.com/issues/61762

    qa: wait_for_clean: failed before timeout expired

* https://tracker.ceph.com/issues/61775

    cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests)

* https://tracker.ceph.com/issues/44565

    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())

* https://tracker.ceph.com/issues/61790

    cephfs client to mds comms remain silent after reconnect

* https://tracker.ceph.com/issues/61791

    snaptest-git-ceph.sh test timed out (job dead)

h3. 20 June 2023

https://pulpito.ceph.com/vshankar-2023-06-15_04:58:28-fs-wip-vshankar-testing-20230614.124123-testing-default-smithi/

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/54462

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/58340

  mds: fsstress.sh hangs with multimds

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/61400