Bug #61225: TestClsRbd.mirror_snapshot failure - rbd - Ceph

Actions

Copy link

Bug #61225

closed

TestClsRbd.mirror_snapshot failure

Added by Laura Flores about 1 year ago. Updated 9 months ago.

Status:

Rejected

Priority:

Normal

Assignee:

Ilya Dryomov

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276082

2023-05-17T01:08:35.448 INFO:tasks.workunit.client.0.smithi077.stdout:[ RUN      ] TestClsRbd.mirror_snapshot
2023-05-17T01:08:35.454 INFO:tasks.workunit.client.0.smithi077.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6-336-g933fa52e/rpm/el8/BUILD/ceph-17.2.6-336-g933fa52e/src/test/cls_rbd/test_cls_rbd.cc:2278: Failure
2023-05-17T01:08:35.454 INFO:tasks.workunit.client.0.smithi077.stdout:Expected equality of these values:
2023-05-17T01:08:35.455 INFO:tasks.workunit.client.0.smithi077.stdout:  -85
2023-05-17T01:08:35.455 INFO:tasks.workunit.client.0.smithi077.stdout:  mirror_image_snapshot_unlink_peer(&ioctx, oid, 1, "peer2")
2023-05-17T01:08:35.455 INFO:tasks.workunit.client.0.smithi077.stdout:    Which is: 0
2023-05-17T01:08:35.455 INFO:tasks.workunit.client.0.smithi077.stdout:[  FAILED  ] TestClsRbd.mirror_snapshot (6 ms)

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276237

First seen on this test card: https://trello.com/c/1EFSeXDn/1752-wip-yuri10-testing-2023-05-16-1243

Related issues 5 (0 open — 5 closed)

Actions

Copy link

Updated by Laura Flores about 1 year ago

Hey Ilya, mind having a look?

Actions

Copy link

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276201
/a/yuriw-2023-05-11_15:01:38-rados-wip-yuri8-testing-2023-05-10-1402-distro-default-smithi/7271220
/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270598

Failed in a few more test branches, so it's not from any of the PRs in the card I linked above. This seems like it's coming from something already merged to main.

Actions

Copy link

Updated by Ilya Dryomov about 1 year ago

Hi Laura,

This is coming from https://github.com/ceph/ceph/pull/51166 and is expected since the change was deliberate. The failure should go away once https://github.com/ceph/ceph/pull/51428 lands in quincy.

Actions

Copy link

Updated by Ilya Dryomov about 1 year ago

Related to Bug #59393: rbd-mirror cannot grab lock in time with latency added

Actions

Copy link

Updated by Ilya Dryomov about 1 year ago

Status changed from New to Rejected

Actions

Copy link

Updated by Laura Flores about 1 year ago

Thanks for looking Ilya! Question: How will a change to quincy affect the main branch in this scenario?

Actions

Copy link

Updated by Ilya Dryomov about 1 year ago

This is an upgrade test: the failure is caused by running a test binary from the quincy branch against OSDs from the main branch. A quincy backport, when it lands, would "catch up" the affected test case.

Actions

Copy link

Updated by Laura Flores about 1 year ago

Ilya Dryomov wrote:

This is an upgrade test: the failure is caused by running a test binary from the quincy branch against OSDs from the main branch. A quincy backport, when it lands, would "catch up" the affected test case.

This non-upgrade test also failed: http://pulpito.front.sepia.ceph.com/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276201/

Does the same logic apply?

Actions

Copy link

Updated by Ilya Dryomov about 1 year ago

This is an "old clients" test, it's using a test binary from pacific. So yes, the same logic applies.

Actions

Copy link

#10

Updated by Laura Flores about 1 year ago

Thanks Ilya. Noted that this issue will be resolved; I will still record instances found in test runs for tracking purposes.

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270634

Actions

Copy link

#11

Updated by Yuri Weinstein 12 months ago

Also see this in many reef runs for example here:

https://pulpito.ceph.com/teuthology-2023-05-20_14:23:11-upgrade:pacific-x-reef-distro-default-smithi/

do we need a backport to reef as well?

Actions

Copy link

#12

Updated by Ilya Dryomov 12 months ago

Yuri Weinstein wrote:

Also see this in many reef runs for example here:

https://pulpito.ceph.com/teuthology-2023-05-20_14:23:11-upgrade:pacific-x-reef-distro-default-smithi/

do we need a backport to reef as well?

It's already backported to reef in https://github.com/ceph/ceph/pull/51518. What is missing in this case is a pacific backport (since this suite is running a test binary from the pacific branch).

Actions

Copy link

#13

Updated by Yuri Weinstein 12 months ago

Ilya Dryomov wrote:

Yuri Weinstein wrote:

Also see this in many reef runs for example here:

https://pulpito.ceph.com/teuthology-2023-05-20_14:23:11-upgrade:pacific-x-reef-distro-default-smithi/

do we need a backport to reef as well?

It's already backported to reef in https://github.com/ceph/ceph/pull/51518. What is missing in this case is a pacific backport (since this suite is running a test binary from the pacific branch).

Then for quincy as well https://pulpito.ceph.com/teuthology-2023-05-20_14:23:11-upgrade:quincy-x-reef-distro-default-smithi/

Actions

Copy link

#14