Actions
main branch¶
2024-04-20¶
https://tracker.ceph.com/issues/65596
- qa: logrotate fails when state file is already locked
- valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space
- qa: error during scrub thrashing: rank damage found: {'backtrace'}
- qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log
- pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
- qa/cephfs: test_cephfs_mirror_blocklist raises KeyError: 'rados_inst'
- qa: health warning no active mgr (MGR_DOWN) occurs before and after test_nfs runs
- qa: test_cd_with_args failure
- qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
- test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
- qa/cephfs: test_multifs_single_path_rootsquash
- client: resends request to same MDS it just received a forward from if it does not have an open session with the target
- pybind/mgr/snap_schedule: 1m scheduled snaps not reliably executed
- qa: fsstress: cannot execute binary file: Exec format error
- qa: untar_snap_rm failure during mds thrashing
- dbench: write failed on handle 10009
- workunits/fsx.sh failure
- ffsb.sh failure Resource temporarily unavailable
2024-04-12¶
https://tracker.ceph.com/issues/65324
(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` and infra issues failures in this run)
- Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
- pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
- qa: ceph tell 4.3a deep-scrub command not found
- qa: scrub - object missing on disk; some files may be lost
- qa: failed cephfs-shell test_reading_conf
2024-04-04¶
https://tracker.ceph.com/issues/65300
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240330.172700
(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` failures in this run)
- Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
- pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
- qa: failed cephfs-shell test_reading_conf
- centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons
- qa: scrub - object missing on disk; some files may be lost
- upgrade stalls after upgrading one ceph-mgr daemon
- qa: test_max_items_per_obj open procs not fully cleaned up
- QA failure: test_fscrypt_dummy_encryption_with_quick_group
- qa/cephfs: test_multifs_single_path_rootsquash
- qa: test_cd_with_args failure
- valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space
4 Apr 2024¶
- https://tracker.ceph.com/issues/64927
qa/cephfs: test_cephfs_mirror_blocklist raises "KeyError: 'rados_inst'" - https://tracker.ceph.com/issues/65022
qa: test_max_items_per_obj open procs not fully cleaned up - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/65136
QA failure: test_fscrypt_dummy_encryption_with_quick_group - https://tracker.ceph.com/issues/65246
qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)
- https://tracker.ceph.com/issues/58945
qa: xfstests-dev's generic test suite has failures with fuse client - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/63265
qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/48562
qa: scrub - object missing on disk; some files may be lost - https://tracker.ceph.com/issues/65020
qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log - https://tracker.ceph.com/issues/64572
workunits/fsx.sh failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/64502
client: ceph-fuse fails to unmount after upgrade to main - https://tracker.ceph.com/issues/54741
crash: MDSTableClient::got_journaled_ack(unsigned long)
- https://tracker.ceph.com/issues/65265
qa: health warning "no active mgr (MGR_DOWN)" occurs before and after test_nfs runs - https://tracker.ceph.com/issues/65308
qa: fs was offline but also unexpectedly degraded - https://tracker.ceph.com/issues/65309
qa: dbench.sh failed with "ERROR: handle 10318 was not found"
- https://tracker.ceph.com/issues/65018
PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)" - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2024-04-02¶
https://tracker.ceph.com/issues/65215
- qa: error during scrub thrashing: rank damage found: {'backtrace'}
- qa: ceph tell 4.3a deep-scrub command not found
- pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
- Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
- workunits/fsx.sh failure
- qa: failed cephfs-shell test_reading_conf
- centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons
- qa: test_max_items_per_obj open procs not fully cleaned up
- qa: dbench workload timeout
- suites/fsstress.sh hangs on one client - test times out
- qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) in cluster log
- qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log
- qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
- ffsb.sh failure Resource temporarily unavailable
- QA failure: test_fscrypt_dummy_encryption_with_quick_group
- qa: cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled in cluster log
- qa: test_cephfs_mirror_cancel_sync fails in a 100 jobs run of fs:mirror suite
2024-03-28¶
https://tracker.ceph.com/issues/65213
- qa: error during scrub thrashing: rank damage found: {'backtrace'}
- workunits/fsx.sh failure
- PG_DEGRADED warnings during cluster creation via cephadm: Health check failed: Degraded data
- suites/fsstress.sh hangs on one client - test times out
- qa: ceph tell 4.3a deep-scrub command not found
- qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
- qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log
- qa: failed cephfs-shell test_reading_conf
- Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
- qa: test_max_items_per_obj open procs not fully cleaned up
- pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
- centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons
- qa: Health check failed: Reduced data availability: 1 pg peering
- qa: dbench workload timeout
2024-03-25¶
https://pulpito.ceph.com/mchangir-2024-03-22_09:46:06-fs:upgrade-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/- https://tracker.ceph.com/issues/64502
fusermount -u fails with: teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds
- https://tracker.ceph.com/issues/62245
libcephfs/test.sh failed - https://tracker.ceph.com/issues/62245#note-3
2024-03-20¶
https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742
https://github.com/batrick/ceph/commit/360516069d9393362c4cc6eb9371680fe16d66ab
Ubuntu jobs filtered out because builds were skipped by jenkins/shaman.
This run has a lot more failures because https://github.com/ceph/ceph/pull/55455 fixed log WRN/ERR checks.
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/64572
workunits/fsx.sh failure - https://tracker.ceph.com/issues/65018
PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)" - https://tracker.ceph.com/issues/64707 (new issue)
suites/fsstress.sh hangs on one client - test times out - https://tracker.ceph.com/issues/64988
qa: fs:workloads mgr client evicted indicated by "cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds" - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/64972
qa: "ceph tell 4.3a deep-scrub" command not found - https://tracker.ceph.com/issues/54108
qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" - https://tracker.ceph.com/issues/65019
qa/suites/fs/top: [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log - https://tracker.ceph.com/issues/65020
qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log - https://tracker.ceph.com/issues/65021
qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/64711
Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/65022
qa: test_max_items_per_obj open procs not fully cleaned up
14th March 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240307.013758
(pjd.sh failures are related to a bug in the testing kernel. See - https://tracker.ceph.com/issues/64679#note-4)
- https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/64502
pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main - https://tracker.ceph.com/issues/64572
workunits/fsx.sh failure - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
5th March 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240304.042522
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/64502
pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/64572
workunits/fsx.sh failure - https://tracker.ceph.com/issues/64707 (new issue)
suites/fsstress.sh hangs on one client - test times out - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/64711
Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/64729 (new issue)
mon.a (mon.0) 1281 : cluster 3 [WRN] MDS_SLOW_METADATA_IO: 3 MDSs report slow metadata IOs" in cluster log - https://tracker.ceph.com/issues/64730
fs/misc/multiple_rsync.sh workunit times out
26th Feb 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240216.060239
(This run is a bit messy due to
a) OCI runtime issues in the testing kernel with centos9
b) SELinux denials related failures
c) Unrelated MON_DOWN warnings)- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/64172
Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/64572
workunits/fsx.sh failure
20th Feb 2024¶
https://github.com/ceph/ceph/pull/55601
https://github.com/ceph/ceph/pull/55659
- https://tracker.ceph.com/issues/64502
client: quincy ceph-fuse fails to unmount after upgrade to main
This run has numerous problems. #55601 introduces testing for the upgrade sequence from reef/{v18.2.0,v18.2.1,reef} as well as an extra dimension for the ceph-fuse client. The main "big" issue is i64502: the ceph-fuse client is not being unmounted when fusermount -u is called. Instead, the client begins to unmount only after daemons are shut down during test cleanup.
19th Feb 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240217.015652
- https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/63764
Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) - https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/64482
ceph: stderr Error: OCI runtime error: crun: bpf create ``: Function not implemented
29 Jan 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240119.075157-1
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/64172
Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) - https://tracker.ceph.com/issues/63265
qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/59684
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/64209
snaptest-multiple-capsnaps.sh fails with "got remote process result: 1"
17th Jan 2024¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240103.072409-1
- https://tracker.ceph.com/issues/63764
Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/63949
leak in mds.c detected by valgrind during CephFS QA run - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/63259
mds: failed to store backtrace and force file system read-only - https://tracker.ceph.com/issues/63265
qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'
16 Jan 2024¶
https://pulpito.ceph.com/rishabh-2023-12-11_15:37:57-fs-rishabh-2023dec11-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-12-17_11:19:43-fs-rishabh-2023dec11-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2024-01-04_18:43:16-fs-rishabh-2024jan4-testing-default-smithi
- https://tracker.ceph.com/issues/63764
Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'}
- https://tracker.ceph.com/issues/63949
valgrind leak in MDS - https://tracker.ceph.com/issues/64041
qa/cephfs: fs/upgrade/nofs suite attempts to jump more than 2 releases - fsstress failure in last run was due a kernel MM layer failure, unrelated to CephFS
- from last run, job #7507400 failed due to MGR; FS wasn't degraded, so it's unrelated to CephFS
06 Dec 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818-x (rerun w/ squid kickoff changes)
- https://tracker.ceph.com/issues/63764
Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/62580
testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout - https://tracker.ceph.com/issues/63265
qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1' - https://tracker.ceph.com/issues/63806
ffsb.sh workunit failure (MDS: std::out_of_range, damaged)
30 Nov 2023¶
- https://tracker.ceph.com/issues/63699
qa: failed cephfs-shell test_reading_conf - https://tracker.ceph.com/issues/63700
qa: test_cd_with_args failure
29 Nov 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231107.042705
- https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/62510 (pending RHEL back port) snaptest-git-ceph.sh failure with fs/thrash
- https://tracker.ceph.com/issues/62810
Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again
14 Nov 2023
(Milind)¶
- https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/63521
qa: Test failure: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/62580
testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/63522
No module named 'tasks.ceph_fuse'
No module named 'tasks.kclient'
No module named 'tasks.cephfs.fuse_mount'
No module named 'tasks.ceph' - https://tracker.ceph.com/issues/63523
Command failed - qa/workunits/fs/misc/general_vxattrs.sh
14 Nov 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231106.073650
(nvm the fs:upgrade test failure - the PR is excluded from merge)
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/62580
testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/63519
ceph-fuse: reef ceph-fuse crashes with main branch ceph-mds - https://tracker.ceph.com/issues/57087
qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure - https://tracker.ceph.com/issues/58945
qa: xfstests-dev's generic test suite has 20 failures with fuse client
7 Nov 2023¶
fs: https://pulpito.ceph.com/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/
re-run: https://pulpito.ceph.com/rishabh-2023-11-05_14:10:09-fs-rishabh-2023nov3-testing-default-smithi/
smoke: https://pulpito.ceph.com/rishabh-2023-11-08_08:39:05-smoke-rishabh-2023nov3-testing-default-smithi/
- https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'}
- https://tracker.ceph.com/issues/63473
fsstress.sh failed with errno 124
3 Nov 2023¶
https://pulpito.ceph.com/rishabh-2023-10-27_06:26:52-fs-rishabh-2023oct26-testing-default-smithi/
- https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/57656
dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'}
- https://tracker.ceph.com/issues/59531
"OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
24 October 2023¶
https://pulpito.ceph.com/?branch=wip-batrick-testing-20231024.144545
Two failures:
https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/
https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/
probably related to https://github.com/ceph/ceph/pull/53255. Killing the mount as part of the test did not complete. Will research more.
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/59531
"OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/63411
qa: flush journal may cause timeouts of `scrub status` - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/63141
test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails
18 Oct 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/59531
"OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/62658
error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds - https://tracker.ceph.com/issues/62580
testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/62036
src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty()) - https://tracker.ceph.com/issues/58945
qa: xfstests-dev's generic test suite has 20 failures with fuse client - https://tracker.ceph.com/issues/62847
mds: blogbench requests stuck (5mds+scrub+snaps-flush)
13 Oct 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/62936
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/47292
cephfs-shell: test_df_for_valid_file failure - https://tracker.ceph.com/issues/63141
qa/cephfs: test_idem_unaffected_root_squash fails - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout - https://tracker.ceph.com/issues/58945
qa: xfstests-dev's generic test suite has 20 failures with fuse client - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS
16 Oct 2023¶
https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825
Infrastructure issues:- /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log
Host lost.
Failures:
- https://tracker.ceph.com/issues/56694
qa: avoid blocking forever on hung umount - https://tracker.ceph.com/issues/63089
qa: tasks/mirror times out - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/59531
"OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/62658
error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/57656
dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/63233
mon|client|mds: valgrind reports possible leaks in the MDS - https://tracker.ceph.com/issues/63278
kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs)
9 Oct 2023¶
https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/
- https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/63141
test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails - https://tracker.ceph.com/issues/62937
logrotate doesn't support parallel execution on same set of logfiles - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55805
error during scrub thrashing reached max tries in 900 secs
26 Sep 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/62873
qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/62682
mon: no mdsmap broadcast after "fs set joinable" is set to true - https://tracker.ceph.com/issues/63089
qa: tasks/mirror times out
22 Sep 2023¶
- https://tracker.ceph.com/issues/59348
qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59531
"OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/61574
build failure for mdtest project - https://tracker.ceph.com/issues/62702
fsstress.sh: MDS slow requests for the internal 'rename' requests - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'}
- https://tracker.ceph.com/issues/62863
deadlock in ceph-fuse causes teuthology job to hang and fail - https://tracker.ceph.com/issues/62870
test_cluster_info (tasks.cephfs.test_nfs.TestNFS) - https://tracker.ceph.com/issues/62873
test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
20 Sep 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout - https://tracker.ceph.com/issues/62658
error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds - https://tracker.ceph.com/issues/62915
qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/62873
qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/62936
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) - https://tracker.ceph.com/issues/62937
Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout - https://tracker.ceph.com/issues/62126
test failure: suites/blogbench.sh stops running - https://tracker.ceph.com/issues/62682
mon: no mdsmap broadcast after "fs set joinable" is set to true
19 Sep 2023¶
- https://tracker.ceph.com/issues/58220#note-9
workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure - https://tracker.ceph.com/issues/62702
Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/62873
qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/59413
cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/62482
qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)
13 Sep 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/61243
qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed - https://tracker.ceph.com/issues/62567
postgres workunit times out - MDS_SLOW_REQUEST in logs - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout
2023 Sep 12¶
A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably:
- Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes.
Failures:
- https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/57656
dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/55805
error scrub thrashing reached max tries in 900 secs - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror - https://tracker.ceph.com/issues/62832
common: config_proxy deadlock during shutdown (and possibly other times) - https://tracker.ceph.com/issues/59413
cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/62567
Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bashc 'sudo -u postgres -pgbench -s 500 -i'" - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58220#note-9
workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/62847
mds: blogbench requests stuck (5mds+scrub+snaps-flush) - https://tracker.ceph.com/issues/62848
qa: fail_fs upgrade scenario hanging - https://tracker.ceph.com/issues/62081
tasks/fscrypt-common does not finish, timesout
11 Sep 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/61243
qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed - https://tracker.ceph.com/issues/62567
postgres workunit times out - MDS_SLOW_REQUEST in logs
6 Sep 2023 Run 2¶
- https://tracker.ceph.com/issues/51964
test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59348
test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/61892
test_strays.TestStrays.test_snapshot_remove failed - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/62484
qa: ffsb.sh test failure - https://tracker.ceph.com/issues/62567
Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bashc 'sudo -u postgres -pgbench -s 500 -i'"
- https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55805
error scrub thrashing reached max tries in 900 secs
6 Sep 2023¶
- https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/61892
test_snapshot_remove (test_strays.TestStrays) failed - https://tracker.ceph.com/issues/59348
qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/62556
test_acls: xfstests_dev: python2 is missing - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/57656
dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
- https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55805
error scrub thrashing reached max tries in 900 secs
- https://tracker.ceph.com/issues/62567
Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bashc 'sudo -u postgres -pgbench -s 500 -i'" - https://tracker.ceph.com/issues/62702
workunit test suites/fsstress.sh on smithi066 with status 124
5 Sep 2023¶
https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/
orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/
this run has failures but acc to Adam King these are not relevant and should be ignored
- https://tracker.ceph.com/issues/61892
test_snapshot_remove (test_strays.TestStrays) failed - https://tracker.ceph.com/issues/59348
test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable" - https://tracker.ceph.com/issues/57656
dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/62187
iozone.sh: line 5: iozone: command not found
- https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55805
error scrub thrashing reached max tries in 900 secs
31 Aug 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/62187
iozone: command not found - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59413
cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" - https://tracker.ceph.com/issues/62653
qa: unimplemented fcntl command: 1036 with fsstress - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/62658
error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds - https://tracker.ceph.com/issues/62188
AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
25 Aug 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807
- https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/62484
qa: ffsb.sh test failure - https://tracker.ceph.com/issues/59531
quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash
24 Aug 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash - https://tracker.ceph.com/issues/62484
qa: ffsb.sh test failure - https://tracker.ceph.com/issues/57087
qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/62187
iozone: command not found - https://tracker.ceph.com/issues/62188
AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) - https://tracker.ceph.com/issues/62567
postgres workunit times out - MDS_SLOW_REQUEST in logs
22 Aug 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61243
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/62188
AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test) - https://tracker.ceph.com/issues/62510
snaptest-git-ceph.sh failure with fs/thrash - https://tracker.ceph.com/issues/62511
src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down)
14 Aug 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/59684 [kclient bug]
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/61243 (NEW)
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
28 JULY 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59531
cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://github.com/ceph/ceph/pull/52556
task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4) - https://tracker.ceph.com/issues/62187
iozone: command not found - https://tracker.ceph.com/issues/61399
qa: build failure for ior (the failed instance is when compiling `mdtest`) - https://tracker.ceph.com/issues/62188
AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
24 Jul 2023¶
https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/
There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures -
https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
One more extra run to check if blogbench.sh fail every time:
https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing -
https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/
- https://tracker.ceph.com/issues/61892
test_snapshot_remove (test_strays.TestStrays) failed - https://tracker.ceph.com/issues/53859
test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/61982
test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/48773
reached max tries: scrub does not complete - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim
- https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10010 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/61399
ior build failure - https://tracker.ceph.com/issues/57676
error during scrub thrashing: backtrace
- https://tracker.ceph.com/issues/38452
'sudou postgres -pgbench -s 500 -i' failed - https://tracker.ceph.com/issues/62126
blogbench.sh failure
18 July 2023¶
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/61182
cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds - https://tracker.ceph.com/issues/61957
test_client_limits.TestClientLimits.test_client_release_bug - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61892
test_strays.TestStrays.test_snapshot_remove failed - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/44565
src/mds/SimpleLock.h: 528: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE || state LOCK_XLOCKSNAP || state LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) - https://tracker.ceph.com/issues/62067
ffsb.sh failure "Resource temporarily unavailable"
17 July 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136
- https://tracker.ceph.com/issues/61982
Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/61182
cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds - https://tracker.ceph.com/issues/61957
test_client_limits.TestClientLimits.test_client_release_bug - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61892
test_strays.TestStrays.test_snapshot_remove failed - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/62036
src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty()) - https://tracker.ceph.com/issues/61737
coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific' - https://tracker.ceph.com/issues/44565
src/mds/SimpleLock.h: 528: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE || state LOCK_XLOCKSNAP || state LOCK_LOCK_XLOCK || state LOCK_LOCK || is_locallock())
13 July 2023 Run 2¶
https://pulpito.ceph.com/rishabh-2023-07-08_23:33:40-fs-wip-rishabh-2023Jul9-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/
- https://tracker.ceph.com/issues/61957
test_client_limits.TestClientLimits.test_client_release_bug - https://tracker.ceph.com/issues/61982
Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/61399
ior build failure
13 July 2023¶
- https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61945
LibCephFS.DelegTimeout failure - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/61982
Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
13 Jul 2023¶
https://pulpito.ceph.com/rishabh-2023-07-05_22:21:20-fs-wip-rishabh-2023Jul5-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-07-06_19:33:28-fs-wip-rishabh-2023Jul5-testing-default-smithi/
- https://tracker.ceph.com/issues/61957
test_client_limits.TestClientLimits.test_client_release_bug - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/48773
scrub does not complete: reached max tries - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/58742
xfstests-dev: kcephfs: generic - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror
12 July 2023¶
https://pulpito.ceph.com/rishabh-2023-07-05_18:32:52-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-07-06_19:46:43-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
- https://tracker.ceph.com/issues/61892
test_strays.TestStrays.test_snapshot_remove failed - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/58742
xfstests-dev: kcephfs: generic - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61182
cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default - https://tracker.ceph.com/issues/48773
reached max tries: scrub does not complete
05 July 2023¶
- https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
27 Jun 2023¶
https://pulpito.ceph.com/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/
- https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/61831
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
22 June 2023¶
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61394
qa/quincy: cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log - https://tracker.ceph.com/issues/61762
qa: wait_for_clean: failed before timeout expired - https://tracker.ceph.com/issues/61775
cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests) - https://tracker.ceph.com/issues/44565
src/mds/SimpleLock.h: 528: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE || state LOCK_XLOCKSNAP || state LOCK_LOCK_XLOCK || state LOCK_LOCK || is_locallock()) - https://tracker.ceph.com/issues/61790
cephfs client to mds comms remain silent after reconnect - https://tracker.ceph.com/issues/61791
snaptest-git-ceph.sh test timed out (job dead)
20 June 2023¶
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/44565
src/mds/SimpleLock.h: 528: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE || state LOCK_XLOCKSNAP || state LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock()) - https://tracker.ceph.com/issues/61737
coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'
16 June 2023¶
https://pulpito.ceph.com/rishabh-2023-05-16_10:39:13-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-05-17_11:09:48-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2023-05-18_10:01:53-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
(bins were rebuilt with a subset of orig PRs) https://pulpito.ceph.com/rishabh-2023-06-09_10:19:22-fs-wip-rishabh-2023Jun9-1308-testing-default-smithi/
- https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59346
fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/61399
libmpich: undefined references to fi_strerror - https://tracker.ceph.com/issues/58945
xfstests-dev: ceph-fuse: generic - https://tracker.ceph.com/issues/58742
xfstests-dev: kcephfs: generic
24 May 2023¶
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59683
Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests - https://tracker.ceph.com/issues/61399
qa: "[Makefile:299: ior] Error 1" - https://tracker.ceph.com/issues/61265
qa: tasks.cephfs.fuse_mount:process failed to terminate after unmount - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/59346
qa/workunits/fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" - https://tracker.ceph.com/issues/61400
valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59344
qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" - https://tracker.ceph.com/issues/61407
mds: abort on CInode::verify_dirfrags - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/61409
qa: _test_stale_caps does not wait for file flush before stat
15 May 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020-6
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59684 [kclient bug]
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/61148
dbench test results in call trace in dmesg [kclient bug] - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds
11 May 2023¶
- https://tracker.ceph.com/issues/59684 [kclient bug]
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55805
error during scrub thrashing reached max tries in 900 secs - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/58220
Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: - https://tracker.ceph.com/issues/58220#note-9
workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure - https://tracker.ceph.com/issues/59342
qa/workunits/kernel_untar_build.sh failed when compiling the Linux source - https://tracker.ceph.com/issues/58949
test_cephfs.test_disk_quota_exceeeded_error - AssertionError: DiskQuotaExceeded not raised by write - https://tracker.ceph.com/issues/61243 (NEW)
test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
11 May 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.054005
(no fsstress job failure [https://tracker.ceph.com/issues/58340] since https://github.com/ceph/ceph/pull/49553
was included in the branch, however, the PR got updated and needs retest).
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/59683
Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests - https://tracker.ceph.com/issues/59684 [kclient bug]
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
09 May 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230506.143554
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/59350
qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR - https://tracker.ceph.com/issues/59683
Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests - https://tracker.ceph.com/issues/59684 [kclient bug]
Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) - https://tracker.ceph.com/issues/59348
qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
10 Apr 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230330.105356
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure
31 Mar 2023¶
run: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-11_05:54:03-fs-wip-rishabh-2023Mar10-1727-testing-default-smithi/
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-23_08:27:28-fs-wip-rishabh-2023Mar20-2250-testing-default-smithi/
There were many more re-runs for "failed+dead" jobs as well as for individual jobs. half of the PRs from the batch were removed (gradually over subsequent re-runs).
- https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58220
Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58220#note-9
workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure - https://tracker.ceph.com/issues/56695
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/58564
workuit dbench failed with error code 1 - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim - https://tracker.ceph.com/issues/57580
Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) - https://tracker.ceph.com/issues/58940
ceph osd hit ceph_abort - https://tracker.ceph.com/issues/55805
error scrub thrashing reached max tries in 900 secs
30 March 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230315.085747
- https://tracker.ceph.com/issues/58938
qa: xfstests-dev's generic test suite has 7 failures with kclient - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds
29 March 2023¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230317.095222
- https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/57087
qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/59230
Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) - https://tracker.ceph.com/issues/58938
qa: xfstests-dev's generic test suite has 7 failures with kclient
13 Mar 2023¶
- https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
09 Mar 2023¶
https://pulpito.ceph.com/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2023-03-08_15:12:36-fs-wip-vshankar-testing-20230308.112059-testing-default-smithi/
- https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/57087
qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
07 Mar 2023¶
https://pulpito.ceph.com/vshankar-2023-03-02_09:21:58-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2023-03-07_05:15:12-fs-wip-vshankar-testing-20230307.030510-testing-default-smithi/
- https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/58220
Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58934
snaptest-git-ceph.sh failure with ceph-fuse
28 Feb 2023¶
- https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
(teuthology infra issues causing testing delays - merging PRs which have tests passing)
25 Jan 2023¶
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim - https://tracker.ceph.com/issues/58220
Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/56011
fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
30 JAN 2023¶
run: http://pulpito.front.sepia.ceph.com/rishabh-2022-11-28_08:04:11-fs-wip-rishabh-testing-2022Nov24-1818-testing-default-smithi/
re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
re-run of re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/55332
Failure in snaptest-git-ceph.sh - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds - https://tracker.ceph.com/issues/58219
Command crashed: 'ceph-dencoder type inode_backtrace_t import - decode dump_json'
- "Failed to load ceph-mgr modules: prometheus" in cluster log"
http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7134086
Acc to Venky this was fixed in https://github.com/ceph/ceph/commit/cf6089200d96fc56b08ee17a4e31f19823370dc8 - Created https://tracker.ceph.com/issues/58564
workunit test suites/dbench.sh failed error code 1
15 Dec 2022¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221215.112736
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/58219
- https://tracker.ceph.com/issues/57655
- qa: fs:mixed-clients kernel_untar_build failure
Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/58340
mds: fsstress.sh hangs with multimds
08 Dec 2022¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221130.043104
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221209.043803
(lots of transient git.ceph.com failures)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/57655
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/58219
Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) - https://tracker.ceph.com/issues/58220
Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/58244
Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan)
14 Oct 2022¶
https://pulpito.ceph.com/vshankar-2022-10-12_04:56:59-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-10-14_04:04:57-fs-wip-vshankar-testing-20221014-072608-testing-default-smithi/
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/57682
client: ERROR: test_reconnect_after_blocklisted - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
10 Oct 2022¶
reruns- fs-thrash, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-04_13:19:47-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
- fs-verify, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-05_12:25:37-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
- cephadm failures also passed after many re-runs: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-06_13:50:51-fs-wip-rishabh-testing-30Sep2022-2-testing-default-smithi/ ** needed this PR to be merged in ceph-ci branch - https://github.com/ceph/ceph/pull/47458
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke - https://tracker.ceph.com/issues/57299
qa: test_dump_loads fails with JSONDecodeError - https://tracker.ceph.com/issues/57655 [Exist in main as well]
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim
2022 Sep 29¶
- https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/36593
Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/56632
Test failure: test_subvolume_snapshot_clone_quota_exceeded - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing
2022 Sep 26¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220923.171109
- https://tracker.ceph.com/issues/55804
qa failure: pjd link tests failed - https://tracker.ceph.com/issues/57676
qa: error during scrub thrashing: rank damage found: {'backtrace'} - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57580
Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/57299
qa: test_dump_loads fails with JSONDecodeError - https://tracker.ceph.com/issues/57280
qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman - https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/57677
qa: "1 MDSs behind on trimming (MDS_TRIM)" - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim - https://tracker.ceph.com/issues/57446
qa: test_subvolume_snapshot_info_if_orphan_clone fails - https://tracker.ceph.com/issues/57655 [Exist in main as well]
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57682
client: ERROR: test_reconnect_after_blocklisted
2022 Sep 22¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220920.234701
- https://tracker.ceph.com/issues/57299
qa: test_dump_loads fails with JSONDecodeError - https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57580
Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) - https://tracker.ceph.com/issues/57280
qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim - https://tracker.ceph.com/issues/51267
CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
NEW:
- https://tracker.ceph.com/issues/57656
[testing] dbench: write failed on handle 10009 (Resource temporarily unavailable) - https://tracker.ceph.com/issues/57655 [Exist in main as well]
qa: fs:mixed-clients kernel_untar_build failure - https://tracker.ceph.com/issues/57657
mds: scrub locates mismatch between child accounted_rstats and self rstats
Segfault probably caused by: https://github.com/ceph/ceph/pull/47795#issuecomment-1255724799
2022 Sep 16¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220905-132828
- https://tracker.ceph.com/issues/57446
qa: test_subvolume_snapshot_info_if_orphan_clone fails - https://tracker.ceph.com/issues/57299
qa: test_dump_loads fails with JSONDecodeError - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57280
qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/48203
https://tracker.ceph.com/issues/36593
qa: quota failure
qa: quota failure caused by clients stepping on each other - https://tracker.ceph.com/issues/57580
Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
2022 Aug 26¶
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-22_17:49:59-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-24_11:56:51-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
- https://tracker.ceph.com/issues/57206
libcephfs/test.sh: ceph_test_libcephfs_reclaim - https://tracker.ceph.com/issues/56632
Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
- https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/36593
Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke)
2022 Aug 22¶
https://pulpito.ceph.com/vshankar-2022-08-12_09:34:24-fs-wip-vshankar-testing1-20220812-072441-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-08-18_04:30:42-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/ (drop problematic PR and re-run)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/51278
mds: "FAILED ceph_assert(!segments.empty())" - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/57205
Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) - https://tracker.ceph.com/issues/57206
ceph_test_libcephfs_reclaim crashes during test - https://tracker.ceph.com/issues/53859
Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke)
2022 Aug 12¶
https://pulpito.ceph.com/vshankar-2022-08-10_04:06:00-fs-wip-vshankar-testing-20220805-190751-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-08-11_12:16:58-fs-wip-vshankar-testing-20220811-145809-testing-default-smithi/ (drop problematic PR and re-run)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
2022 Aug 04¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220804-123835 (only mgr/volumes, mgr/stats)
Unrealted teuthology failure on rhel
2022 Jul 25¶
1st re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_03:51:19-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi
2nd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
3rd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
4th (final) re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-28_03:59:01-fs-wip-rishabh-testing-2022Jul28-0143-testing-default-smithi/
- https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke)
- https://tracker.ceph.com/issues/54460
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/36593
Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1 - https://tracker.ceph.com/issues/54462
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128~
2022 July 22¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220721.235756
MDS_HEALTH_DUMMY error in log fixed by followup commit.
transient selinux ping failure
- https://tracker.ceph.com/issues/56694
qa: avoid blocking forever on hung umount - https://tracker.ceph.com/issues/56695
[RHEL stock] pjd test failures - https://tracker.ceph.com/issues/56696
admin keyring disappears during qa run - https://tracker.ceph.com/issues/56697
qa: fs/snaps fails for fuse - https://tracker.ceph.com/issues/50222
osd: 5.2s0 deep-scrub : stat mismatch - https://tracker.ceph.com/issues/56698
client: FAILED ceph_assert(_size == 0) - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)"
2022 Jul 15¶
- https://tracker.ceph.com/issues/53859
Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) - https://tracker.ceph.com/issues/50223
client.xxxx isn't responding to mclientcaps(revoke) - https://tracker.ceph.com/issues/50222
osd: deep-scrub : stat mismatch
- https://tracker.ceph.com/issues/56632
Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) - https://tracker.ceph.com/issues/56634
workunit test fs/snaps/snaptest-intodir.sh - https://tracker.ceph.com/issues/56644
Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022 July 05¶
On 1st re-run some jobs passed - http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/
On 2nd re-run only few jobs failed -
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
- https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/55804
Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdirp -/home/ubuntu/cephtest/
- https://tracker.ceph.com/issues/56445
Command failed on smithi080 with status 123: "find /home/ubuntu/cephtest/archive/syslogname '*.log' -print0 | sudo xargs -0 --no-run-if-empty -gzip --" - https://tracker.ceph.com/issues/51267
Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdirp -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 - https://tracker.ceph.com/issues/50224
Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
2022 July 04¶
https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/
(rhel runs were borked due to: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/JSZQFUKVLDND4W33PXDGCABPHNSPT6SS/, tests ran with --filter-out=rhel)
- https://tracker.ceph.com/issues/56445
Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslogname '*.log' -print0 | sudo xargs -0 --no-run-if-empty -gzip --" - https://tracker.ceph.com/issues/56446
Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2022 June 20¶
https://pulpito.ceph.com/vshankar-2022-06-15_04:03:39-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-06-19_08:22:46-fs-wip-vshankar-testing1-20220619-102531-testing-default-smithi/
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/55804
qa failure: pjd link tests failed - https://tracker.ceph.com/issues/54108
qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" - https://tracker.ceph.com/issues/55332
Failure in snaptest-git-ceph.sh (it's an async unlink/create bug)
2022 June 13¶
- https://tracker.ceph.com/issues/56024
cephadm: removes ceph.conf during qa run causing command failure - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/56012
mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
2022 Jun 13¶
https://pulpito.ceph.com/vshankar-2022-06-07_00:25:50-fs-wip-vshankar-testing-20220606-223254-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-06-10_01:04:46-fs-wip-vshankar-testing-20220609-175550-testing-default-smithi/
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/55804
qa failure: pjd link tests failed - https://tracker.ceph.com/issues/56003
client: src/include/xlist.h: 81: FAILED ceph_assert(_size == 0) - https://tracker.ceph.com/issues/56011
fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison - https://tracker.ceph.com/issues/56012
mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
2022 Jun 07¶
https://pulpito.ceph.com/vshankar-2022-06-06_21:25:41-fs-wip-vshankar-testing1-20220606-230129-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-06-07_10:53:31-fs-wip-vshankar-testing1-20220607-104134-testing-default-smithi/ (rerun after dropping a problematic PR)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure
2022 May 12¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220509-125847
https://pulpito.ceph.com/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/ (drop prs + rerun)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/55332
Failure in snaptest-git-ceph.sh - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/55538
Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) - https://tracker.ceph.com/issues/55258
lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs (cropss up again, though very infrequent)
2022 May 04¶
https://pulpito.ceph.com/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/ (after dropping PRs)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/55332
Failure in snaptest-git-ceph.sh - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) - https://tracker.ceph.com/issues/55516
qa: fs suite tests failing with "json.decoder.JSONDecodeError: Extra data: line 2 column 82 (char 82)" - https://tracker.ceph.com/issues/55537
mds: crash during fs:upgrade test - https://tracker.ceph.com/issues/55538
Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2022 Apr 25¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220420-113951 (owner vshankar)
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/55258
lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs - https://tracker.ceph.com/issues/55377
kclient: mds revoke Fwb caps stuck after the kclient tries writebcak once
2022 Apr 14¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220411-144044
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/55170
mds: crash during rejoin (CDir::fetch_keys) - https://tracker.ceph.com/issues/55331
pjd failure - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/55332
Failure in snaptest-git-ceph.sh - https://tracker.ceph.com/issues/55258
lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
2022 Apr 11¶
https://pulpito.ceph.com/?branch=wip-vshankar-testing-55110-20220408-203242
- https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/48680
mds: scrubbing stuck "scrub active (0 inodes in the stack)" - https://tracker.ceph.com/issues/55236
qa: fs/snaps tests fails with "hit max job timeout" - https://tracker.ceph.com/issues/54108
qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" - https://tracker.ceph.com/issues/54971
Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/55258
lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
2022 Mar 21¶
Run didn't go well, lots of failures - debugging by dropping PRs and running against master branch. Only merging unrelated PRs that pass tests.
2022 Mar 08¶
rerun with
- (drop) https://github.com/ceph/ceph/pull/44679
- (drop) https://github.com/ceph/ceph/pull/44958
https://pulpito.ceph.com/vshankar-2022-03-06_14:47:51-fs-wip-vshankar-testing-20220304-132102-testing-default-smithi/
- https://tracker.ceph.com/issues/54419 (new)
`ceph orch upgrade start` seems to never reach completion - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing
2022 Feb 09¶
rerun with
- (drop) https://github.com/ceph/ceph/pull/37938
- (drop) https://github.com/ceph/ceph/pull/44335
- (drop) https://github.com/ceph/ceph/pull/44491
- (drop) https://github.com/ceph/ceph/pull/44501
https://pulpito.ceph.com/vshankar-2022-02-08_14:27:29-fs-wip-vshankar-testing-20220208-181241-testing-default-smithi/
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/54066
test_subvolume_no_upgrade_v1_sanity fails with `AssertionError: 1000 != 0` - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout
2022 Feb 01¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220127.171526
- https://tracker.ceph.com/issues/54107
kclient: hang during umount - https://tracker.ceph.com/issues/54106
kclient: hang during workunit cleanup - https://tracker.ceph.com/issues/54108
qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/52438
qa: ffsb timeout
2022 Jan 13¶
rerun with:
- (add) https://github.com/ceph/ceph/pull/44570
- (drop) https://github.com/ceph/ceph/pull/43184
https://pulpito.ceph.com/vshankar-2022-01-13_04:42:40-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/
- https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/53859
qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2022 Jan 03¶
https://pulpito.ceph.com/vshankar-2021-12-22_07:37:44-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2022-01-03_12:27:45-fs-wip-vshankar-testing-20220103-142738-testing-default-smithi/ (rerun)
- https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/51267
CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:... - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/51278
mds: "FAILED ceph_assert(!segments.empty())" - https://tracker.ceph.com/issues/52279
cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter
2021 Dec 22¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211222.014316
- https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52279
cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure - https://tracker.ceph.com/issues/48773
qa: scrub does not complete
2021 Nov 30¶
https://pulpito.ceph.com/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/
https://pulpito.ceph.com/vshankar-2021-11-30_06:23:32-fs-wip-vshankar-testing-20211124-094330-distro-default-smithi/ (rerun w/ QA fixes)
- https://tracker.ceph.com/issues/53436
mds, mon: mds beacon messages get dropped? (mds never reaches up:active state) - https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/48812
qa: test_scrub_pause_and_resume_with_abort failure - https://tracker.ceph.com/issues/51076
"wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend. - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2021 November 9¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211109.180315
- https://tracker.ceph.com/issues/53214
qa: "dd: error reading '/sys/kernel/debug/ceph/2a934501-6731-4052-a836-f42229a869be.client4874/metrics': Is a directory" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/53216
qa: "RuntimeError: value of attributes should be either str or None. client_id" - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2021 November 03¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211103.023355
- https://tracker.ceph.com/issues/51964
qa: test_cephfs_mirror_restart_sync_on_blocklist failure - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/52436
fs/ceph: "corrupt mdsmap" - https://tracker.ceph.com/issues/53074
pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active - https://tracker.ceph.com/issues/53150
pybind/mgr/cephadm/upgrade: tolerate MDS failures during upgrade straddling v16.2.5 - https://tracker.ceph.com/issues/53155
MDSMonitor: assertion during upgrade to v16.2.5+
2021 October 26¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211025.000447
- https://tracker.ceph.com/issues/53074
pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active - https://tracker.ceph.com/issues/52997
testing: hang ing umount - https://tracker.ceph.com/issues/50824
qa: snaptest-git-ceph bus error - https://tracker.ceph.com/issues/52436
fs/ceph: "corrupt mdsmap" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/53082
ceph-fuse: segmenetation fault in Client::handle_mds_map - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2021 October 19¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211019.013028
- https://tracker.ceph.com/issues/52995
qa: test_standby_count_wanted failure - https://tracker.ceph.com/issues/52948
osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up" - https://tracker.ceph.com/issues/52996
qa: test_perf_counters via test_openfiletable - https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80 - https://tracker.ceph.com/issues/52997
testing: hang ing umount - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/48773
qa: scrub does not complete
2021 October 12¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211012.192211
Some failures caused by teuthology bug: https://tracker.ceph.com/issues/52944
New test caused failure: https://github.com/ceph/ceph/pull/43297#discussion_r729883167
- https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/52948
osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure - https://tracker.ceph.com/issues/52949
RuntimeError: The following counters failed to be set on mds daemons: {'mds.dir_split'}
2021 October 02¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211002.163337
Some failures caused by cephadm upgrade test. Fixed in follow-up qa commit.
test_simple failures caused by PR in this set.
A few reruns because of QA infra noise.
- https://tracker.ceph.com/issues/52822
qa: failed pacific install on fs:upgrade - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete
2021 September 20¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210917.174826
- https://tracker.ceph.com/issues/52677
qa: test_simple failure - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch) - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/52438
qa: ffsb timeout
2021 September 10¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210910.181451
- https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") - https://tracker.ceph.com/issues/52624
qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" - https://tracker.ceph.com/issues/52625
qa: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) - https://tracker.ceph.com/issues/52439
qa: acls does not compile on centos stream - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/52626
mds: ScrubStack.cc: 831: FAILED ceph_assert(diri) - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch)
2021 August 27¶
Several jobs died because of device failures.
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210827.024746
- https://tracker.ceph.com/issues/52430
mds: fast async create client mount breaks racy test - https://tracker.ceph.com/issues/52436
fs/ceph: "corrupt mdsmap" - https://tracker.ceph.com/issues/52437
mds: InoTable::replay_release_ids abort via test_inotable_sync - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/52438
qa: ffsb timeout - https://tracker.ceph.com/issues/52439
qa: acls does not compile on centos stream
2021 July 30¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210729.214022
- https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51975
pybind/mgr/stats: KeyError
2021 July 28¶
with qa fix: https://pulpito.ceph.com/pdonnell-2021-07-28_16:20:28-fs-wip-pdonnell-testing-20210728.141004-distro-basic-smithi/
- https://tracker.ceph.com/issues/51905
qa: "error reading sessionmap 'mds1_sessionmap'" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones") - https://tracker.ceph.com/issues/51267
CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:... - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch)
2021 July 16¶
- https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80 - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch) - https://tracker.ceph.com/issues/50824
qa: snaptest-git-ceph bus error
2021 July 04¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210703.052904
- https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/39150
mon: "FAILED ceph_assert(session_map.sessions.empty())" when out of quorum - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch) - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2021 July 01¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210701.192056
- https://tracker.ceph.com/issues/51197
qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details - https://tracker.ceph.com/issues/50866
osd: stat mismatch on objects - https://tracker.ceph.com/issues/48773
qa: scrub does not complete
2021 June 26¶
- https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/51410
kclient: fails to finish reconnect during MDS thrashing (testing branch) - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80
2021 June 21¶
One failure caused by PR: https://github.com/ceph/ceph/pull/41935#issuecomment-866472599
- https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings - https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing - https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/50495
libcephfs: shutdown race fails with status 141 - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/50824
qa: snaptest-git-ceph bus error - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)"
2021 June 16¶
MDS abort class of failures caused by PR: https://github.com/ceph/ceph/pull/41667
- https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/43216
MDSMonitor: removes MDS coming out of quorum election - https://tracker.ceph.com/issues/51278
mds: "FAILED ceph_assert(!segments.empty())" - https://tracker.ceph.com/issues/51279
kclient hangs on umount (testing branch) - https://tracker.ceph.com/issues/51280
mds: "FAILED ceph_assert(r 0 || r -2)" - https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/51281
qa: snaptest-snap-rm-cmp.sh: "echo 'FAIL: bad match, /tmp/a 4637e766853d1ad16a7b17079e2c6f03 != real c3883760b18d50e8d78819c54d579b00'" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51076
"wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend. - https://tracker.ceph.com/issues/51228
qa: rmdir: failed to remove 'a/.snap/*': No such file or directory - https://tracker.ceph.com/issues/51282
pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2021 June 14¶
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
- https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/51228
qa: rmdir: failed to remove 'a/.snap/*': No such file or directory - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/51182
pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' - https://tracker.ceph.com/issues/51229
qa: test_multi_snap_schedule list difference failure - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing
2021 June 13¶
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
- https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51182
pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' - https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/51197
qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2021 June 11¶
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
- https://tracker.ceph.com/issues/51169
qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing - https://tracker.ceph.com/issues/43216
MDSMonitor: removes MDS coming out of quorum election - https://tracker.ceph.com/issues/51182
pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs' - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/51183
qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions' - https://tracker.ceph.com/issues/51184
qa: fs:bugs does not specify distro
2021 June 03¶
- https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/49845#note-2 (regression)
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/43216
MDSMonitor: removes MDS coming out of quorum election
2021 May 18¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.214114
Regression in testing kernel caused some failures. Ilya fixed those and rerun
looked better. Some odd new noise in the rerun relating to packaging and "No
module named 'tasks.ceph'".
- https://tracker.ceph.com/issues/50824
qa: snaptest-git-ceph bus error - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/49845#note-2 (regression)
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/48203 (stock kernel update required)
qa: quota failure
2021 May 18¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.025642
- https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/45591
mgr: FAILED ceph_assert(daemon != nullptr) - https://tracker.ceph.com/issues/50866
osd: stat mismatch on objects - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" - https://tracker.ceph.com/issues/50867
qa: fs:mirror: reduced data availability - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50868
qa: "kern.log.gz already exists; not overwritten" - https://tracker.ceph.com/issues/50870
qa: test_full: "rm: cannot remove 'large_file_a': Permission denied"
2021 May 11¶
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210511.232042
- one class of failures caused by PR
- https://tracker.ceph.com/issues/48812
qa: test_scrub_pause_and_resume_with_abort failure - https://tracker.ceph.com/issues/50390
mds: monclient: wait_auth_rotating timed out after 30 - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/50825
qa: snaptest-git-ceph hang during mon thrashing v2 - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50823
qa: RuntimeError: timeout waiting for cluster to stabilize
2021 May 14¶
https://pulpito.ceph.com/pdonnell-2021-05-14_21:45:42-fs-master-distro-basic-smithi/
- https://tracker.ceph.com/issues/48812
qa: test_scrub_pause_and_resume_with_abort failure - https://tracker.ceph.com/issues/50821
qa: untar_snap_rm failure during mds thrashing - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/50822
qa: testing kernel patch for client metrics causes mds abort - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50823
qa: RuntimeError: timeout waiting for cluster to stabilize - https://tracker.ceph.com/issues/50824
qa: snaptest-git-ceph bus error - https://tracker.ceph.com/issues/50825
qa: snaptest-git-ceph hang during mon thrashing v2 - https://tracker.ceph.com/issues/50826
kceph: stock RHEL kernel hangs on snaptests with mon|osd thrashers
2021 May 01¶
- https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/50281
qa: untar_snap_rm timeout - https://tracker.ceph.com/issues/48203 (stock kernel update required)
qa: quota failure - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50390
mds: monclient: wait_auth_rotating timed out after 30 - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/50622 (regression)
msg: active_connections regression - https://tracker.ceph.com/issues/45591
mgr: FAILED ceph_assert(daemon != nullptr) - https://tracker.ceph.com/issues/50221
qa: snaptest-git-ceph failure in git diff - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2021 Apr 15¶
- https://tracker.ceph.com/issues/50281
qa: untar_snap_rm timeout - https://tracker.ceph.com/issues/50220
qa: dbench workload timeout - https://tracker.ceph.com/issues/50246
mds: failure replaying journal (EMetaBlob) - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" - https://tracker.ceph.com/issues/50222
osd: 5.2s0 deep-scrub : stat mismatch - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/49845
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/37808
osd: osdmap cache weak_refs assert during shutdown - https://tracker.ceph.com/issues/50387
client: fs/snaps failure - https://tracker.ceph.com/issues/50389
mds: "cluster [ERR] Error recovering journal 0x203: (2) No such file or directory" in cluster log" - https://tracker.ceph.com/issues/50216
qa: "ls: cannot access 'lost+found': No such file or directory" - https://tracker.ceph.com/issues/50390
mds: monclient: wait_auth_rotating timed out after 30
2021 Apr 08¶
- https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/50279
qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c" - https://tracker.ceph.com/issues/50246
mds: failure replaying journal (EMetaBlob) - https://tracker.ceph.com/issues/48365
qa: ffsb build failure on CentOS 8.2 - https://tracker.ceph.com/issues/50216
qa: "ls: cannot access 'lost+found': No such file or directory" - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50280
cephadm: RuntimeError: uid/gid not found - https://tracker.ceph.com/issues/50281
qa: untar_snap_rm timeout
2021 Apr 08¶
https://pulpito.ceph.com/pdonnell-2021-04-08_04:31:36-fs-wip-pdonnell-testing-20210408.024225-distro-basic-smithi/
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210408.142238 (with logic inversion / QA fix)
- https://tracker.ceph.com/issues/50246
mds: failure replaying journal (EMetaBlob) - https://tracker.ceph.com/issues/50250
mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
2021 Apr 07¶
- https://tracker.ceph.com/issues/50215
qa: "log [ERR] : error reading sessionmap 'mds2_sessionmap'" - https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" - https://tracker.ceph.com/issues/50216
qa: "ls: cannot access 'lost+found': No such file or directory" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/49845
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/50220
qa: dbench workload timeout - https://tracker.ceph.com/issues/50221
qa: snaptest-git-ceph failure in git diff - https://tracker.ceph.com/issues/50222
osd: 5.2s0 deep-scrub : stat mismatch - https://tracker.ceph.com/issues/50223
qa: "client.4737 isn't responding to mclientcaps(revoke)" - https://tracker.ceph.com/issues/50224
qa: test_mirroring_init_failure_with_recovery failure
2021 Apr 01¶
- https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80 - https://tracker.ceph.com/issues/50177
osd: "stalled aio... buggy kernel or bad device?" - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing - https://tracker.ceph.com/issues/49845
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/50178
qa: "TypeError: run() got an unexpected keyword argument 'shell'" - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2021 Mar 24¶
- https://tracker.ceph.com/issues/49500
qa: "Assertion `cb_done' failed." - https://tracker.ceph.com/issues/50019
qa: mount failure with cephadm "probably no MDS server is up?" - https://tracker.ceph.com/issues/50020
qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80 - https://tracker.ceph.com/issues/50021
qa: snaptest-git-ceph failure during mon thrashing - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing - https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes" - https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2021 Mar 18¶
- https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/49845
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/49605
mgr: drops command on the floor - https://tracker.ceph.com/issues/48203 (stock kernel update required)
qa: quota failure - https://tracker.ceph.com/issues/49928
client: items pinned in cache preventing unmount x2
2021 Mar 15¶
- https://tracker.ceph.com/issues/49842
qa: stuck pkg install - https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" - https://tracker.ceph.com/issues/49822
test: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) failure - https://tracker.ceph.com/issues/49240
terminate called after throwing an instance of 'std::bad_alloc' - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/49500
qa: "Assertion `cb_done' failed." - https://tracker.ceph.com/issues/49843
qa: fs/snaps/snaptest-upchildrealms.sh failure - https://tracker.ceph.com/issues/49845
qa: failed umount in test_volumes - https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/49605
mgr: drops command on the floor
and failure caused by PR: https://github.com/ceph/ceph/pull/39969
2021 Mar 09¶
- https://tracker.ceph.com/issues/49500
qa: "Assertion `cb_done' failed." - https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details" - https://tracker.ceph.com/issues/48773
qa: scrub does not complete - https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed - https://tracker.ceph.com/issues/49240
terminate called after throwing an instance of 'std::bad_alloc' - https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" - https://tracker.ceph.com/issues/49684
qa: fs:cephadm mount does not wait for mds to be created - https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing
Updated by Patrick Donnelly 3 days ago · 261 revisions