Project

General

Profile

Actions

Bug #44938

closed

[rbd-mirror] tx-only peer from heartbeat can race w/ CLI

Added by Jason Dillaman about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If a peer cluster's rbd-mirror sends a heartbeat it will create a tx-only peer entry. However, this can race w/ the rbd CLI in the tests when it tries to add the same peer as a rx-tx peer.

http://qa-proxy.ceph.com/teuthology/jdillaman-2020-04-03_13:25:28-rbd-wip-jd-testing-distro-basic-smithi/4922729/teuthology.log


Related issues 1 (0 open1 closed)

Copied to rbd - Backport #45036: octopus: [rbd-mirror] tx-only peer from heartbeat can race w/ CLIResolvedNathan CutlerActions
Actions #1

Updated by Jason Dillaman about 4 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 34422
Actions #2

Updated by Mykola Golub about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #3

Updated by Nathan Cutler about 4 years ago

  • Copied to Backport #45036: octopus: [rbd-mirror] tx-only peer from heartbeat can race w/ CLI added
Actions #4

Updated by Jason Dillaman about 4 years ago

NOTE: this will require a second commit to address since the first commit did not fully resolve the issue.

http://qa-proxy.ceph.com/teuthology/jdillaman-2020-04-14_08:24:43-rbd-wip-jd-testing-2-distro-basic-smithi/4953440/teuthology.log

Actions #5

Updated by Jason Dillaman about 4 years ago

  • Status changed from Pending Backport to Fix Under Review
Actions #6

Updated by Mykola Golub about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #8

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions #9

Updated by Jason Dillaman almost 4 years ago

  • Status changed from Resolved to Pending Backport
Actions #10

Updated by Nathan Cutler almost 4 years ago

@Jason Borden The octopus backport PR for this issue was already merged. Is it missing something?

Actions #11

Updated by Jason Dillaman almost 4 years ago

Yes, see comment #7. It has an additional fix that we should backport. I'm just re-using these tracker tickets instead of opening new ones.

Actions #12

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions #13

Updated by Nathan Cutler over 3 years ago

Jason Dillaman wrote:

Yes, see comment #7. It has an additional fix that we should backport. I'm just re-using these tracker tickets instead of opening new ones.

I believe both commits are now in octopus:

commit f93516d78734740c30c5532e5032be1893004bb1
Author: Mykola Golub <mgolub@suse.com>
Date:   Sat Apr 25 08:36:25 2020 +0100

    qa/workunits/rbd: retry the addition of a mirror pool peer

    fb4311f5 has fixed this for setup, but "remove mirroring pool" 
    test needs fixing too.

    Fixes: https://tracker.ceph.com/issues/44938
    Signed-off-by: Mykola Golub <mgolub@suse.com>
    (cherry picked from commit 7eced158a9a3c47cc408b35219b4428e97e018fb)

commit 4644cd663de27bd19b07eb8dca0153032060694b
Author: Jason Dillaman <dillaman@redhat.com>
Date:   Wed Apr 15 16:27:07 2020 -0400

    qa/workunits/rbd: retry the addition of a mirror pool peer

    We might race with the remote rbd-mirror daemon creating a
    tx-only peer when adding a new peer. Therefore, delete the
    tx-only peer and attempt to re-create it.

    Fixes: https://tracker.ceph.com/issues/44938
    Signed-off-by: Jason Dillaman <dillaman@redhat.com>
    (cherry picked from commit fb4311f597a98b6870d7895e6403fb32356bfbe9)
Actions

Also available in: Atom PDF