Quincy » History » Revision 40
Revision 39 (Milind Changire, 10/17/2022 04:52 AM) → Revision 40/90 (Milind Changire, 10/21/2022 06:30 AM)
h1. Quincy h2. On-call Schedule * Feb: Patrick * Mar: Jeff * Apr: Jos Collin * May: Ramana * Jun: Xiubo * Jul: Rishabh * Aug: Kotresh * Sep: Venky * Oct: Milind h3. h2. 2022 Oct 21 http://pulpito.front.sepia.ceph.com/yuriw-2022-10-17_17:37:24-fs-wip-yuri-testing-2022-10-17-0746-quincy-distro-default-smithi/ * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57446 Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) * https://tracker.ceph.com/issues/55825 cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log h3. 2022 Oct 17 http://pulpito.front.sepia.ceph.com/yuriw-2022-10-12_16:32:23-fs-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/ * https://tracker.ceph.com/issues/54460 snaptest-multiple-capsnaps.sh test failure * https://tracker.ceph.com/issues/57446 Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) * https://tracker.ceph.com/issues/55825 cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 cluster [WRN] client.4490 isn't responding to mclientcaps(revoke) h3. h2. 2022 Sep 29 http://pulpito.front.sepia.ceph.com/?branch=wip-yuri6-testing-2022-09-23-1008-quincy * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57446 qa: test_subvolume_snapshot_info_if_orphan_clone fails * https://tracker.ceph.com/issues/50224 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/57280 qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) h3. h2. 2022 Sep 09 http://pulpito.front.sepia.ceph.com/yuriw-2022-09-08_18:29:21-fs-wip-yuri6-testing-2022-09-08-0859-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/51282 cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/57205 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) * https://tracker.ceph.com/issues/57446 qa: test_subvolume_snapshot_info_if_orphan_clone fails * https://tracker.ceph.com/issues/51964 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/57280 Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing h3. h2. 2022 Sep 02 https://pulpito.ceph.com/yuriw-2022-09-01_18:27:02-fs-wip-yuri11-testing-2022-09-01-0804-quincy-distro-default-smithi/ and https://pulpito.ceph.com/?branch=wip-lflores-testing-2-2022-08-26-2240-quincy * https://tracker.ceph.com/issues/57280 Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/48773 error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 h3. h2. 2022 Aug 31 https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-08-23-1120-quincy * https://tracker.ceph.com/issues/51964 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/57280 Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/48773 error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) h3. h2. 2022 Aug 17 https://pulpito.ceph.com/yuriw-2022-08-17_18:46:04-fs-wip-yuri7-testing-2022-08-17-0943-quincy-distro-default-smithi/ There were following errors not related to tests which fixed in rerun: * Command failed on smithi161 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c ... -- bash -c 'ceph fs dump'" * Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing * reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up * https://tracker.ceph.com/issues/56697 - qa: fs/snaps fails for fuse - Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp ..." * SSH connection to smithi077 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' Rerun: https://pulpito.ceph.com/yuriw-2022-08-18_15:08:53-fs-wip-yuri7-testing-2022-08-17-0943-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/51282 cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log h3. h2. 2022 Aug 10 http://pulpito.front.sepia.ceph.com/yuriw-2022-08-11_02:21:28-fs-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/ * Most of the failures are passed in re-run. Please check rerun failures below. - tasks/{1-thrash/mon 2-workunit/fs/snaps - reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up - tasks/{1-thrash/osd 2-workunit/suites/iozone - reached maximum tries (90) after waiting for 540 seconds - DEBUG:teuthology.misc:7 of 8 OSDs are up - tasks/metrics - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - tasks/scrub - No module named 'tasks.cephfs.fuse_mount' - tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} - No module named 'tasks.fs' - tasks/snap-schedule - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log - tasks/volumes/{overrides test/clone}} - No module named 'tasks.ceph' - tasks/snapshots - CommandFailedError: Command failed on smithi035 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes - INFO:teuthology.orchestra.run.smithi035.stderr:E: Version '17.2.3-414-ge5c30ac2-1focal' for 'python-ceph' was not found - INFO:teuthology.orchestra.run.smithi035.stderr:E: Unable to locate package libcephfs1 - tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} - No module named 'tasks.ceph' - tasks/{1-thrash/osd 2-workunit/suites/pjd}} - No module named 'tasks.ceph' - tasks/cfuse_workunit_suites_fsstress traceless/50pc} - No module named 'tasks' - tasks/{0-octopus 1-upgrade}} - No module named 'tasks' - tasks/{1-thrash/osd 2-workunit/fs/snaps}} - cluster [WRN] client.4520 isn't responding to mclientcaps(revoke), - tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} - reached maximum tries (90) after waiting for 540 seconds - teuthology.misc:7 of 8 OSDs are up Re-run1: http://pulpito.front.sepia.ceph.com/yuriw-2022-08-11_14:24:26-fs-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/ * tasks/{1-thrash/mon 2-workunit/fs/snaps - reached maximum tries (90) after waiting for 540 seconds DEBUG:teuthology.misc:7 of 8 OSDs are up * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) * tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} - reached maximum tries (90) after waiting for 540 seconds DEBUG:teuthology.misc:7 of 8 OSDs are up Re-run2: http://pulpito.front.sepia.ceph.com/yuriw-2022-08-16_14:46:15-fs-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log h3. h2. 2022 Aug 03 https://pulpito.ceph.com/yuriw-2022-08-04_11:54:20-fs-wip-yuri8-testing-2022-08-03-1028-quincy-distro-default-smithi/ Re-run: https://pulpito.ceph.com/yuriw-2022-08-09_15:36:21-fs-wip-yuri8-testing-2022-08-03-1028-quincy-distro-default-smithi * No module named 'tasks' - Fixed in re-run * https://tracker.ceph.com/issues/51282 cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log * https://tracker.ceph.com/issues/57064 qa: test_add_ancestor_and_child_directory failure * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 cluster [WRN] client.xxxx isn't responding to mclientcaps(revoke) h3. h2. 2022 Jul 22 https://pulpito.ceph.com/yuriw-2022-07-11_13:37:40-fs-wip-yuri5-testing-2022-07-06-1020-quincy-distro-default-smithi/ re-run: https://pulpito.ceph.com/yuriw-2022-07-12_13:37:44-fs-wip-yuri5-testing-2022-07-06-1020-quincy-distro-default-smithi/ Most failure weren't seen in re-run. * http://tracker.ceph.com/issues/52624 Health check failed: Reduced data availability * https://tracker.ceph.com/issues/50223 client.xxxx isn't responding to mclientcaps(revoke) * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 h3. h2. 2022 Jul 13 https://pulpito.ceph.com/yuriw-2022-07-08_17:05:01-fs-wip-yuri2-testing-2022-07-08-0453-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/51964 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) * https://tracker.ceph.com/issues/48773 error during scrub thrashing: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' h3. h2. 2022 Jun 08 http://pulpito.front.sepia.ceph.com/yuriw-2022-06-07_22:29:43-fs-wip-yuri3-testing-2022-06-07-0722-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log h3. h2. 2022 Jun 07 http://pulpito.front.sepia.ceph.com/yuriw-2022-06-02_20:32:25-fs-wip-yuri5-testing-2022-06-02-0825-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log h3. h2. 2022 Jun 03 https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-06-02-0810-quincy * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 h3. h2. 2022 May 31 http://pulpito.front.sepia.ceph.com/yuriw-2022-05-27_21:58:39-fs-wip-yuri2-testing-2022-05-27-1033-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log h3. h2. 2022 May 26 https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-05-10-1027-quincy * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 h3. h2. 2022 May 10 http://pulpito.front.sepia.ceph.com/?branch=wip-yuri-testing-2022-05-05-0838-quincy * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" h3. h2. 2022 April 29 https://pulpito.ceph.com/?branch=wip-yuri3-testing-2022-04-22-0534-quincy * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 h3. h2. 2022 April 13 http://pulpito.front.sepia.ceph.com/?branch=wip-yuri3-testing-2022-04-11-0746-quincy * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * https://tracker.ceph.com/issues/52438 qa: ffsb timeout h3. h2. 2022 March 31 http://pulpito.front.sepia.ceph.com/yuriw-2022-03-29_20:09:22-fs-wip-yuri-testing-2022-03-29-0741-quincy-distro-default-smithi/ http://pulpito.front.sepia.ceph.com/yuriw-2022-03-30_14:35:58-fs-wip-yuri-testing-2022-03-29-0741-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/54460 snaptest-multiple-capsnaps.sh test failure * https://tracker.ceph.com/issues/50223 qa: "client.4737 isn't responding to mclientcaps(revoke)" * http://tracker.ceph.com/issues/54606 check-counter task runs till max job timeout Handful of failed jobs due to: <pre> Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c5bb4e7d582f118c1093d94fbfedfb197eaa03b4 -v bootstrap --fsid 44e07f86-b03b-11ec-8c35-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.55 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' </pre> h3. h2. 2022 March 17 http://pulpito.front.sepia.ceph.com/yuriw-2022-03-14_18:57:01-fs-wip-yuri2-testing-2022-03-14-0946-quincy-distro-default-smithi/ * http://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * http://tracker.ceph.com/issues/54461 ffsb.sh test failure * http://tracker.ceph.com/issues/54606 check-counter task runs till max job timeout Couple of jobs that are dead with: <pre> 2022-03-15T05:15:22.447 ERROR:paramiko.transport:Socket exception: No route to host (113) 2022-03-15T05:15:22.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-03-15T05:15:22.453 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... </pre> h3. h2. 2022 March 1 * https://tracker.ceph.com/issues/51282 (maybe?) cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log * https://tracker.ceph.com/issues/52624 cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log * https://tracker.ceph.com/issues/54460 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' * https://tracker.ceph.com/issues/50223 cluster [WRN] client.14480 isn't responding to mclientcaps(revoke), ino 0x1000000f3fd pending pAsLsXsFsc issued pAsLsXsFscb, sent 304.933510 seconds ago" in cluster log * https://tracker.ceph.com/issues/54461 Command failed (workunit test suites/ffsb.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' * https://tracker.ceph.com/issues/54462 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=465157b30605a0c958df893de628c923386baa8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'