Bug #58265
closed
TestClsRbd.group_snap_list_max_read failure during upgrade/parallel tests
Added by Laura Flores over 1 year ago.
Updated about 1 year ago.
Description
/a/yuriw-2022-12-10_00:03:28-rados-wip-yuri7-testing-2022-12-09-1107-quincy-distro-default-smithi/7111235
2022-12-10T02:27:15.101 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read
2022-12-10T02:27:15.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:Dec 10 02:27:15 smithi115 bash[56163]: cluster 2022-12-10T02:27:14.248663+0000 mgr.y (mgr.24967) 669 : cluster [DBG] pgmap v1313: 137 pgs: 137 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 639 KiB/s wr, 326 op/s
2022-12-10T02:27:15.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Dec 10 02:27:15 smithi115 bash[54747]: cluster 2022-12-10T02:27:14.248663+0000 mgr.y (mgr.24967) 669 : cluster [DBG] pgmap v1313: 137 pgs: 137 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 639 KiB/s wr, 326 op/s
2022-12-10T02:27:15.712 INFO:journalctl@ceph.mon.b.smithi137.stdout:Dec 10 02:27:15 smithi137 bash[45906]: cluster 2022-12-10T02:27:14.248663+0000 mgr.y (mgr.24967) 669 : cluster [DBG] pgmap v1313: 137 pgs: 137 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 639 KiB/s wr, 326 op/s
2022-12-10T02:27:16.083 INFO:tasks.workunit.client.0.smithi115.stdout:/build/ceph-16.2.10-1479-g5e1b6cb9/src/test/cls_rbd/test_cls_rbd.cc:2768: Failure
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout:Expected equality of these values:
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout: 150U
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout: Which is: 150
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout: snapshots.size()
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout: Which is: 500
2022-12-10T02:27:16.084 INFO:tasks.workunit.client.0.smithi115.stdout:[ FAILED ] TestClsRbd.group_snap_list_max_read (1058 ms)
/a/yuriw-2022-12-10_00:03:28-rados-wip-yuri7-testing-2022-12-09-1107-quincy-distro-default-smithi/7111391
- Subject changed from TestClsRbd.group_snap_list_max_read failure to TestClsRbd.group_snap_list_max_read failure during upgrade/parallel tests
/a/ksirivad-2022-12-21_15:23:02-rados-wip-ksirivad-test-quincy-distro-default-smithi/7123713/
/a/yuriw-2023-01-21_17:26:05-upgrade:pacific-x-wip-yuri6-testing-2023-01-20-0728-distro-default-smithi/7132591
/a/yuriw-2023-02-03_22:00:43-rados-wip-yuri7-testing-2023-01-30-1510-quincy-distro-default-smithi/7148044
I believe it was failing because TestClsRbd.group_snap_list_max_read test had been merged into pacific much earlier [1] than the cls fix to quincy [2]. The quincy fix was merged just a couple of days ago. I suppose the latest run /a/yuriw-2023-02-03_22:00:43-rados-wip-yuri7-testing-2023-01-30-1510-quincy-distro-default-smithi/7148044 still did not include the fix and hope the problem will gone in the future.
[1] https://github.com/ceph/ceph/pull/49195
[2] https://github.com/ceph/ceph/pull/49196
- Related to Bug #58914: [ FAILED ] TestClsRbd.group_snap_list_max_read in upgrade:quincy-x-reef added
- Status changed from New to Rejected
- Assignee set to Ilya Dryomov
- Backport deleted (
quincy)
Mykola Golub wrote:
I believe it was failing because TestClsRbd.group_snap_list_max_read test had been merged into pacific much earlier [1] than the cls fix to quincy [2]. The quincy fix was merged just a couple of days ago. I suppose the latest run /a/yuriw-2023-02-03_22:00:43-rados-wip-yuri7-testing-2023-01-30-1510-quincy-distro-default-smithi/7148044 still did not include the fix and hope the problem will gone in the future.
Yup, in /a/yuriw-2023-02-03_22:00:43-rados-wip-yuri7-testing-2023-01-30-1510-quincy-distro-default-smithi/7148044 the test was run against a quincy integration branch without https://github.com/ceph/ceph/pull/49196 included:
2023-02-04T02:16:58.066 INFO:teuthology.orchestra.run.smithi191.stdout:osd.0 smithi191 running (6m) 47s ago 13m 675M 2489M 17.2.5-755-ge1fa1cc9 678b6ca032a2 f0dacb343eae
2023-02-04T02:16:58.066 INFO:teuthology.orchestra.run.smithi191.stdout:osd.1 smithi191 running (5m) 47s ago 13m 573M 2489M 17.2.5-755-ge1fa1cc9 678b6ca032a2 afc02f84ef21
2023-02-04T02:16:58.066 INFO:teuthology.orchestra.run.smithi191.stdout:osd.2 smithi191 running (5m) 47s ago 13m 429M 2489M 17.2.5-755-ge1fa1cc9 678b6ca032a2 7c6d576929af
2023-02-04T02:16:58.066 INFO:teuthology.orchestra.run.smithi191.stdout:osd.3 smithi191 running (4m) 47s ago 12m 369M 2489M 17.2.5-755-ge1fa1cc9 678b6ca032a2 4a5c60866530
2023-02-04T02:16:58.067 INFO:teuthology.orchestra.run.smithi191.stdout:osd.4 smithi203 running (3m) 24s ago 12m 593M 2488M 17.2.5-755-ge1fa1cc9 678b6ca032a2 bd48db76d238
2023-02-04T02:16:58.067 INFO:teuthology.orchestra.run.smithi191.stdout:osd.5 smithi203 running (3m) 24s ago 12m 422M 2488M 17.2.5-755-ge1fa1cc9 678b6ca032a2 d8e00b0487ac
2023-02-04T02:16:58.067 INFO:teuthology.orchestra.run.smithi191.stdout:osd.6 smithi203 running (3m) 24s ago 12m 268M 2488M 17.2.5-755-ge1fa1cc9 678b6ca032a2 604710f36472
2023-02-04T02:16:58.067 INFO:teuthology.orchestra.run.smithi191.stdout:osd.7 smithi203 running (2m) 24s ago 11m 340M 2488M 17.2.5-755-ge1fa1cc9 678b6ca032a2 3bb7868b4d3a
$ git log e1fa1cc9ee2edc67a5b1bdd75f5932f385e44a4f | grep 'cls/rbd: update last read in group::snap_list'
$ git log quincy | grep 'cls/rbd: update last read in group::snap_list'
cls/rbd: update last read in group::snap_list
However, there appears to be a bigger issue with these jobs, currently discussed in #58914.
Adding this instance here to document it:
/a/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7224036
Also available in: Atom
PDF