Ceph » CephFS

h1. <code>main</code> branch

h3. 2024-04-30

"wip-pdonnell-testing-20240429.210911-debug":https://tracker.ceph.com/issues/65694

* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676

* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108

* "valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)":https://tracker.ceph.com/issues/65314

* "qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) in cluster log":https://tracker.ceph.com/issues/65021

* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020

* "qa: test_cd_with_args failure":https://tracker.ceph.com/issues/63700

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "qa: Health detail: HEALTH_WARN Degraded data redundancy: 40/348 objects degraded (11.494%), 9 pgs degraded in cluster log":https://tracker.ceph.com/issues/65700

* "qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)":https://tracker.ceph.com/issues/65246

* "qa: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)":https://tracker.ceph.com/issues/53859

* "qa/cephfs: test_cephfs_mirror_blocklist raises KeyError: 'rados_inst'":https://tracker.ceph.com/issues/64927

* "qa/suites/fs/upgrade: Command failed ... ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1":https://tracker.ceph.com/issues/65703

h3. 26 APR 2024

 * https://pulpito.ceph.com/rishabh-2024-04-24_05:22:11-fs-wip-rishabh-testing-20240416.193735-5-testing-default-smithi/

* https://tracker.ceph.com/issues/63700

  qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/64927

  qa/cephfs: test_cephfs_mirror_blocklist raises "KeyError: 'rados_inst'"

* https://tracker.ceph.com/issues/65022

  qa: test_max_items_per_obj open procs not fully cleaned up

* https://tracker.ceph.com/issues/53859

  qa: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/65136

  QA failure: test_fscrypt_dummy_encryption_with_quick_group

* https://tracker.ceph.com/issues/64572

  workunits/fsx.sh failure

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/65265

  qa: health warning "no active mgr (MGR_DOWN)" occurs before and after test_nfs runs

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/64502

  pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/65020

  qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log

* https://tracker.ceph.com/issues/48562

  qa: scrub - object missing on disk; some files may be lost

* https://tracker.ceph.com/issues/55805

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

h3. 2024-04-20

https://tracker.ceph.com/issues/65596

* "qa: logrotate fails when state file is already locked":https://tracker.ceph.com/issues/65612

* "valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)":https://tracker.ceph.com/issues/65314

* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676

* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "qa/cephfs: test_cephfs_mirror_blocklist raises KeyError: 'rados_inst'":https://tracker.ceph.com/issues/64927

* "qa: health warning no active mgr (MGR_DOWN) occurs before and after test_nfs runs":https://tracker.ceph.com/issues/65265

* "qa: test_cd_with_args failure":https://tracker.ceph.com/issues/63700

* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108

* "test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed":https://tracker.ceph.com/issues/61243

* "qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)":https://tracker.ceph.com/issues/65246

* "client: resends request to same MDS it just received a forward from if it does not have an open session with the target":https://tracker.ceph.com/issues/65614

* "pybind/mgr/snap_schedule: 1m scheduled snaps not reliably executed":https://tracker.ceph.com/issues/65616

* "qa: fsstress: cannot execute binary file: Exec format error":https://tracker.ceph.com/issues/65618

* "qa: untar_snap_rm failure during mds thrashing":https://tracker.ceph.com/issues/50821

* "[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)":https://tracker.ceph.com/issues/57656

* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572

* "ffsb.sh failure Resource temporarily unavailable":https://tracker.ceph.com/issues/62067

h3. 2024-04-12

https://tracker.ceph.com/issues/65324

(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` and infra issues failures in this run)

* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972

* "qa: scrub - object missing on disk; some files may be lost":https://tracker.ceph.com/issues/48562

* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699

h3. 2024-04-04

https://tracker.ceph.com/issues/65300

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240330.172700

(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` failures in this run)

* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699

* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774

* "qa: scrub - object missing on disk; some files may be lost":https://tracker.ceph.com/issues/48562

* "upgrade stalls after upgrading one ceph-mgr daemon":https://tracker.ceph.com/issues/65263

* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022

* "QA failure: test_fscrypt_dummy_encryption_with_quick_group":https://tracker.ceph.com/issues/65136

* "qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)":https://tracker.ceph.com/issues/65246

* "qa: test_cd_with_args failure":https://tracker.ceph.com/issues/63700

* "valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)":https://tracker.ceph.com/issues/65314

h3. 4 Apr 2024

https://pulpito.ceph.com/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/

* https://tracker.ceph.com/issues/64927

  qa/cephfs: test_cephfs_mirror_blocklist raises "KeyError: 'rados_inst'"

* https://tracker.ceph.com/issues/65022

  qa: test_max_items_per_obj open procs not fully cleaned up

* https://tracker.ceph.com/issues/63699

  qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/63700

  qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/65136

  QA failure: test_fscrypt_dummy_encryption_with_quick_group

* https://tracker.ceph.com/issues/65246

  qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)

* https://tracker.ceph.com/issues/58945

  qa: xfstests-dev's generic test suite has failures with fuse client

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/63265

  qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/63949

  leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/48562

  qa: scrub - object missing on disk; some files may be lost

* https://tracker.ceph.com/issues/65020

  qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log

* https://tracker.ceph.com/issues/64572

  workunits/fsx.sh failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64502

  client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/54741

  crash: MDSTableClient::got_journaled_ack(unsigned long)

* https://tracker.ceph.com/issues/65265

  qa: health warning "no active mgr (MGR_DOWN)" occurs before and after test_nfs runs

* https://tracker.ceph.com/issues/65308

  qa: fs was offline but also unexpectedly degraded

* https://tracker.ceph.com/issues/65309

  qa: dbench.sh failed with "ERROR: handle 10318 was not found"

* https://tracker.ceph.com/issues/65018

  PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)"

* https://tracker.ceph.com/issues/52624

  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

h3. 2024-04-02

https://tracker.ceph.com/issues/65215

* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676

* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711

* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572

* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699

* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774

* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022

* "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220

* "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707

* "qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) in cluster log":https://tracker.ceph.com/issues/65021

* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020

* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108

* "ffsb.sh failure Resource temporarily unavailable":https://tracker.ceph.com/issues/62067

* "QA failure: test_fscrypt_dummy_encryption_with_quick_group":https://tracker.ceph.com/issues/65136

* "qa: cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled in cluster log":https://tracker.ceph.com/issues/65271

* "qa: test_cephfs_mirror_cancel_sync fails in a 100 jobs run of fs:mirror suite":https://tracker.ceph.com/issues/64534

h3. 2024-03-28

https://tracker.ceph.com/issues/65213

* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676

* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572

* "PG_DEGRADED warnings during cluster creation via cephadm: Health check failed: Degraded data":https://tracker.ceph.com/issues/65018

* "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707

* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972

* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108

* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020

* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699

* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711

* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022

* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502

* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774

* "qa: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)":https://tracker.ceph.com/issues/52624

* "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220

h3. 2024-03-25

https://pulpito.ceph.com/mchangir-2024-03-22_09:46:06-fs:upgrade-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/

* https://tracker.ceph.com/issues/64502

  fusermount -u fails with: teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds

https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/

* https://tracker.ceph.com/issues/62245

  libcephfs/test.sh failed - https://tracker.ceph.com/issues/62245#note-3

h3. 2024-03-20

https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742

https://github.com/batrick/ceph/commit/360516069d9393362c4cc6eb9371680fe16d66ab

Ubuntu jobs filtered out because builds were skipped by jenkins/shaman.

This run has a lot more failures because https://github.com/ceph/ceph/pull/55455 fixed log WRN/ERR checks.

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/65018

    PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)"

* https://tracker.ceph.com/issues/64707 (new issue)

    suites/fsstress.sh hangs on one client - test times out

* https://tracker.ceph.com/issues/64988

    qa: fs:workloads mgr client evicted indicated by "cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds"

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/64972

    qa: "ceph tell 4.3a deep-scrub" command not found

* https://tracker.ceph.com/issues/54108

    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"

* https://tracker.ceph.com/issues/65019

    qa/suites/fs/top: [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 

* https://tracker.ceph.com/issues/65020

    qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log

* https://tracker.ceph.com/issues/65021

    qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64711

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/50821

    qa: untar_snap_rm failure during mds thrashing

* https://tracker.ceph.com/issues/65022

    qa: test_max_items_per_obj open procs not fully cleaned up

h3.  14th March 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240307.013758

(pjd.sh failures are related to a bug in the testing kernel. See - https://tracker.ceph.com/issues/64679#note-4)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64502

    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

h3. 5th March 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240304.042522

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/64502

    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

* https://tracker.ceph.com/issues/64707 (new issue)

    suites/fsstress.sh hangs on one client - test times out

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/64711

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/64729 (new issue)

    mon.a (mon.0) 1281 : cluster 3 [WRN] MDS_SLOW_METADATA_IO: 3 MDSs report slow metadata IOs" in cluster log

* https://tracker.ceph.com/issues/64730

    fs/misc/multiple_rsync.sh workunit times out

h3. 26th Feb 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240216.060239

(This run is a bit messy due to

  a) OCI runtime issues in the testing kernel with centos9

  b) SELinux denials related failures

  c) Unrelated MON_DOWN warnings)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64172

    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/64572

    workunits/fsx.sh failure

h3. 20th Feb 2024

https://github.com/ceph/ceph/pull/55601

https://github.com/ceph/ceph/pull/55659

https://pulpito.ceph.com/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/

* https://tracker.ceph.com/issues/64502

    client: quincy ceph-fuse fails to unmount after upgrade to main

This run has numerous problems. #55601 introduces testing for the upgrade sequence from </code>reef/{v18.2.0,v18.2.1,reef}</code> as well as an extra dimension for the ceph-fuse client. The main "big" issue is i64502: the ceph-fuse client is not being unmounted when <code>fusermount -u</code> is called. Instead, the client begins to unmount only after daemons are shut down during test cleanup.

h3. 19th Feb 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240217.015652

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/64482

    ceph: stderr Error: OCI runtime error: crun: bpf create ``: Function not implemented

h3. 29 Jan 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240119.075157-1

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/64172

    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/59684

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/64209

    snaptest-multiple-capsnaps.sh fails with "got remote process result: 1"

h3. 17th Jan 2024

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240103.072409-1

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63949

    leak in mds.c detected by valgrind during CephFS QA run

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63259

    mds: failed to store backtrace and force file system read-only

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

h3. 16 Jan 2024

https://pulpito.ceph.com/rishabh-2023-12-11_15:37:57-fs-rishabh-2023dec11-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-12-17_11:19:43-fs-rishabh-2023dec11-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2024-01-04_18:43:16-fs-rishabh-2024jan4-testing-default-smithi

* https://tracker.ceph.com/issues/63764

  Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63141

  qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable" 

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/54462 

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63949

  valgrind leak in MDS

* https://tracker.ceph.com/issues/64041

  qa/cephfs: fs/upgrade/nofs suite attempts to jump more than 2 releases

* fsstress failure in last run was due a kernel MM layer failure, unrelated to CephFS

* from last run, job #7507400 failed due to MGR; FS wasn't degraded, so it's unrelated to CephFS

h3. 06 Dec 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818-x (rerun w/ squid kickoff changes)

* https://tracker.ceph.com/issues/63764

    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/63265

    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

* https://tracker.ceph.com/issues/63806

    ffsb.sh workunit failure (MDS: std::out_of_range, damaged)

h3. 30 Nov 2023

https://pulpito.ceph.com/pdonnell-2023-11-30_08:05:19-fs:shell-wip-batrick-testing-20231130.014408-distro-default-smithi/

* https://tracker.ceph.com/issues/63699

    qa: failed cephfs-shell test_reading_conf

* https://tracker.ceph.com/issues/63700

    qa: test_cd_with_args failure

h3. 29 Nov 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231107.042705

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62510 (pending RHEL back port)
    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62810

    Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again

h3. 14 Nov 2023

(Milind)

https://pulpito.ceph.com/mchangir-2023-11-13_10:27:15-fs-wip-mchangir-testing-20231110.052303-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63521

  qa: Test failure: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63522

    No module named 'tasks.ceph_fuse'

    No module named 'tasks.kclient'

    No module named 'tasks.cephfs.fuse_mount'

    No module named 'tasks.ceph'

* https://tracker.ceph.com/issues/63523

    Command failed - qa/workunits/fs/misc/general_vxattrs.sh

h3. 14 Nov 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231106.073650

(nvm the fs:upgrade test failure - the PR is excluded from merge)

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/63519

    ceph-fuse: reef ceph-fuse crashes with main branch ceph-mds

* https://tracker.ceph.com/issues/57087

    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

h3. 7 Nov 2023

fs: https://pulpito.ceph.com/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/

re-run: https://pulpito.ceph.com/rishabh-2023-11-05_14:10:09-fs-rishabh-2023nov3-testing-default-smithi/

smoke: https://pulpito.ceph.com/rishabh-2023-11-08_08:39:05-smoke-rishabh-2023nov3-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63473

  fsstress.sh failed with errno 124

h3. 3 Nov 2023

https://pulpito.ceph.com/rishabh-2023-10-27_06:26:52-fs-rishabh-2023oct26-testing-default-smithi/

* https://tracker.ceph.com/issues/63141

  qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/52624

  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

h3. 24 October 2023

https://pulpito.ceph.com/?branch=wip-batrick-testing-20231024.144545

Two failures:

https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/

https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/

probably related to https://github.com/ceph/ceph/pull/53255. Killing the mount as part of the test did not complete. Will research more.

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/59531

    "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS 

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/63411

    qa: flush journal may cause timeouts of `scrub status`

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/63141

    test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails

h3. 18 Oct 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/62658

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62580

    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62036

    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

* https://tracker.ceph.com/issues/62847

    mds: blogbench requests stuck (5mds+scrub+snaps-flush)

h3. 13 Oct 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62936

    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/47292

    cephfs-shell: test_df_for_valid_file failure

* https://tracker.ceph.com/issues/63141

    qa/cephfs: test_idem_unaffected_root_squash fails

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/58945

    qa: xfstests-dev's generic test suite has 20 failures with fuse client

* https://tracker.ceph.com/issues/63233

    mon|client|mds: valgrind reports possible leaks in the MDS

h3. 16 Oct 2023

https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825

Infrastructure issues:

* /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log

    Host lost.

One followup fix:

* https://pulpito.ceph.com/pdonnell-2023-10-20_00:33:29-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/

Failures:

* https://tracker.ceph.com/issues/56694

    qa: avoid blocking forever on hung umount

* https://tracker.ceph.com/issues/63089

    qa: tasks/mirror times out

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62658 

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/61243

  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/63233

  mon|client|mds: valgrind reports possible leaks in the MDS

* https://tracker.ceph.com/issues/63278

  kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs)

h3. 9 Oct 2023

https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/63141

  test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails

* https://tracker.ceph.com/issues/62937

  logrotate doesn't support parallel execution on same set of logfiles

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error during scrub thrashing reached max tries in 900 secs

h3. 26 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62873

    qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62682

    mon: no mdsmap broadcast after "fs set joinable" is set to true

* https://tracker.ceph.com/issues/63089

    qa: tasks/mirror times out

h3. 22 Sep 2023

https://pulpito.ceph.com/rishabh-2023-09-12_12:12:15-fs-wip-rishabh-2023sep12-b2-testing-default-smithi/

* https://tracker.ceph.com/issues/59348

  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/59531

  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 

* https://tracker.ceph.com/issues/61574

  build failure for mdtest project

* https://tracker.ceph.com/issues/62702

  fsstress.sh: MDS slow requests for the internal 'rename' requests

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62863 

  deadlock in ceph-fuse causes teuthology job to hang and fail

* https://tracker.ceph.com/issues/62870

  test_cluster_info (tasks.cephfs.test_nfs.TestNFS)

* https://tracker.ceph.com/issues/62873

  test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

h3. 20 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/62658 

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62915

    qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/62873

  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/62936

    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

* https://tracker.ceph.com/issues/62937

    Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

* https://tracker.ceph.com/issues/62126

    test failure: suites/blogbench.sh stops running

* https://tracker.ceph.com/issues/62682

    mon: no mdsmap broadcast after "fs set joinable" is set to true

h3. 19 Sep 2023

http://pulpito.front.sepia.ceph.com/mchangir-2023-09-12_05:40:22-fs-wip-mchangir-testing-20230908.140927-testing-default-smithi/

* https://tracker.ceph.com/issues/58220#note-9

  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure

* https://tracker.ceph.com/issues/62702

  Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59348

  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/52624

  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61243

  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/62873

  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)

* https://tracker.ceph.com/issues/59413

  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/62482

  qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)

h3. 13 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909

* https://tracker.ceph.com/issues/52624

      qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

    qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed

* https://tracker.ceph.com/issues/62567

    postgres workunit times out - MDS_SLOW_REQUEST in logs

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

h3. 2023 Sep 12

https://pulpito.ceph.com/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/

A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably:

* Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes.

Failures:

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61399

  libmpich: undefined references to fi_strerror

* https://tracker.ceph.com/issues/62832

  common: config_proxy deadlock during shutdown (and possibly other times)

* https://tracker.ceph.com/issues/59413

  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/62567

  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/58220#note-9

  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/62847

    mds: blogbench requests stuck (5mds+scrub+snaps-flush)

* https://tracker.ceph.com/issues/62848

    qa: fail_fs upgrade scenario hanging

* https://tracker.ceph.com/issues/62081

    tasks/fscrypt-common does not finish, timesout

h3. 11 Sep 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/61243

  qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed

* https://tracker.ceph.com/issues/62567

  postgres workunit times out - MDS_SLOW_REQUEST in logs

h3. 6 Sep 2023 Run 2

https://pulpito.ceph.com/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ 

* https://tracker.ceph.com/issues/51964

  test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59348

  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/61892

  test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/62484

  qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/62567

  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

h3. 6 Sep 2023

https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/

* https://tracker.ceph.com/issues/53859

  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/51964

  qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/59348

  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/54462

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/62556

  test_acls: xfstests_dev: python2 is missing

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57656

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

* https://tracker.ceph.com/issues/62567

  Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"

* https://tracker.ceph.com/issues/62702

  workunit test suites/fsstress.sh on smithi066 with status 124

h3. 5 Sep 2023

https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/

orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/

  this run has failures but acc to Adam King these are not relevant and should be ignored

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/59348

  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota

* https://tracker.ceph.com/issues/54462

  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128

* https://tracker.ceph.com/issues/62067

  ffsb.sh failure "Resource temporarily unavailable"

* https://tracker.ceph.com/issues/57656 

  dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/59346

  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"

* https://tracker.ceph.com/issues/59344

  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"

* https://tracker.ceph.com/issues/50223

  client.xxxx isn't responding to mclientcaps(revoke)

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/62187

  iozone.sh: line 5: iozone: command not found

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/55805

  error scrub thrashing reached max tries in 900 secs

h3. 31 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59413

    cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"

* https://tracker.ceph.com/issues/62653

    qa: unimplemented fcntl command: 1036 with fsstress

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/62658

    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

h3. 25 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62484

    qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/59531

    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

h3. 24 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62484

    qa: ffsb.sh test failure

* https://tracker.ceph.com/issues/57087

    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

* https://tracker.ceph.com/issues/62567

    postgres workunit times out - MDS_SLOW_REQUEST in logs

h3. 22 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/61243

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

* https://tracker.ceph.com/issues/62510

    snaptest-git-ceph.sh failure with fs/thrash

* https://tracker.ceph.com/issues/62511

    src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down)

h3. 14 Aug 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59531

    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/59684 [kclient bug]

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)

* https://tracker.ceph.com/issues/61243 (NEW)

    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/57656

    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)

h3. 28 JULY 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/61399

    ior build failure

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/59531

    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://github.com/ceph/ceph/pull/52556

    task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4)

* https://tracker.ceph.com/issues/62187

    iozone: command not found

* https://tracker.ceph.com/issues/61399

    qa: build failure for ior (the failed instance is when compiling `mdtest`)

* https://tracker.ceph.com/issues/62188

    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)

h3. 24 Jul 2023

https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/

https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/

There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures -

https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/

One more extra run to check if blogbench.sh fail every time:

https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/

blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing -

https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/

* https://tracker.ceph.com/issues/61892

  test_snapshot_remove (test_strays.TestStrays) failed

* https://tracker.ceph.com/issues/53859

  test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

* https://tracker.ceph.com/issues/61982

  test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

* https://tracker.ceph.com/issues/52438

  qa: ffsb timeout

* https://tracker.ceph.com/issues/54460

  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

  qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/48773

  reached max tries: scrub does not complete

* https://tracker.ceph.com/issues/58340

  mds: fsstress.sh hangs with multimds

* https://tracker.ceph.com/issues/61400

  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default

* https://tracker.ceph.com/issues/57206

  libcephfs/test.sh: ceph_test_libcephfs_reclaim

* https://tracker.ceph.com/issues/57656

  [testing] dbench: write failed on handle 10010 (Resource temporarily unavailable)

* https://tracker.ceph.com/issues/61399

  ior build failure

* https://tracker.ceph.com/issues/57676

  error during scrub thrashing: backtrace

* https://tracker.ceph.com/issues/38452

  'sudo -u postgres -- pgbench -s 500 -i' failed

* https://tracker.ceph.com/issues/62126

  blogbench.sh failure

h3. 18 July 2023

* https://tracker.ceph.com/issues/52624

    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 

* https://tracker.ceph.com/issues/57676

    qa: error during scrub thrashing: rank damage found: {'backtrace'}

* https://tracker.ceph.com/issues/54460

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1

* https://tracker.ceph.com/issues/57655

    qa: fs:mixed-clients kernel_untar_build failure

* https://tracker.ceph.com/issues/51964

    qa: test_cephfs_mirror_restart_sync_on_blocklist failure

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61182

    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds

* https://tracker.ceph.com/issues/61957

    test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/59348

    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

* https://tracker.ceph.com/issues/61892

    test_strays.TestStrays.test_snapshot_remove failed

* https://tracker.ceph.com/issues/59346

    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 

* https://tracker.ceph.com/issues/44565

    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())

* https://tracker.ceph.com/issues/62067

    ffsb.sh failure "Resource temporarily unavailable"

h3. 17 July 2023

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136

* https://tracker.ceph.com/issues/61982

    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)

* https://tracker.ceph.com/issues/59344

    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 

* https://tracker.ceph.com/issues/61182

    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds

* https://tracker.ceph.com/issues/61957

    test_client_limits.TestClientLimits.test_client_release_bug

* https://tracker.ceph.com/issues/61400

    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc