Project

General

Profile

Actions

Bug #17993

closed

rbd-mirror: potential race mirroring cloned image

Added by Jason Dillaman over 7 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

With pool-level replication enabled, when a clone is created, it first creates an empty standalone image and adds it to the mirroring directory. If the remote rbd-mirror daemon detects the new image before the clone process completes, it the replication will fail since it will try to create a non-cloned image on the remote side. This is a very small race potential that was discovered after improving new replicated image discovery.


Related issues 2 (0 open2 closed)

Copied to rbd - Backport #18500: jewel: rbd-mirror: potential race mirroring cloned imageRejectedJason DillamanActions
Copied to rbd - Backport #18501: kraken: rbd-mirror: potential race mirroring cloned imageResolvedNathan CutlerActions
Actions #1

Updated by Yang Dongsheng over 7 years ago

Hi Jason,
What about add a key-value in header object to mark this image is ready for mirror or not? such as "mirror_ready = true|false".

Then when we are cloning an image, put an false in mirror_ready until all work finished, when rbd-mirror detect a new image in this pool, rbd-mirror should
check the mirror_ready at first, if false, wait a moment and check mirror_ready again until it's ready or timed out.

Does this sound good?

Jason Dillaman wrote:

With pool-level replication enabled, when a clone is created, it first creates an empty standalone image and adds it to the mirroring directory. If the remote rbd-mirror daemon detects the new image before the clone process completes, it the replication will fail since it will try to create a non-cloned image on the remote side. This is a very small race potential that was discovered after improving new replicated image discovery.

Actions #2

Updated by Jason Dillaman over 7 years ago

We track this state already -- it's just being enabled too soon since clone is re-using the create image state machine. Therefore, the create image state machine just needs a bool to say "don't enable mirroring yet" and then instead the clone state machine will enable mirroring once it has completed setting the parent.

Actions #3

Updated by Yang Dongsheng over 7 years ago

Jason Dillaman wrote:

We track this state already -- it's just being enabled too soon since clone is re-using the create image state machine. Therefore, the create image state machine just needs a bool to say "don't enable mirroring yet" and then instead the clone state machine will enable mirroring once it has completed setting the parent.

Oh, yes, it's in RBD_MIRROR object. Okey, will reuse it. thanx

Actions #4

Updated by Yang Dongsheng over 7 years ago

Yang Dongsheng wrote:

Jason Dillaman wrote:

We track this state already -- it's just being enabled too soon since clone is re-using the create image state machine. Therefore, the create image state machine just needs a bool to say "don't enable mirroring yet" and then instead the clone state machine will enable mirroring once it has completed setting the parent.

Oh, yes, it's in RBD_MIRROR object. Okey, will reuse it. thanx

As I have a blocked pull request about CloneRequest, I will put this after CloneRequest merged.

Actions #5

Updated by Jason Dillaman over 7 years ago

  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman
Actions #6

Updated by Jason Dillaman over 7 years ago

  • Status changed from In Progress to Fix Under Review
Actions #7

Updated by Mykola Golub over 7 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #8

Updated by Loïc Dachary over 7 years ago

  • Copied to Backport #18500: jewel: rbd-mirror: potential race mirroring cloned image added
Actions #9

Updated by Loïc Dachary over 7 years ago

  • Copied to Backport #18501: kraken: rbd-mirror: potential race mirroring cloned image added
Actions #10

Updated by liuzhong chen over 6 years ago

I wonder this patch was signed backport to Jewel but not do it.Is there some problem to backport to jewel or something else?Thank you!

Actions #11

Updated by Nathan Cutler about 6 years ago

I attempted the jewel backport at #18500 but it is beyond my abilities.

Actions #12

Updated by liuzhong chen about 6 years ago

Nathan Cutler wrote:

I attempted the jewel backport at #18500 but it is beyond my abilities.

@Nathan Weinberg OK,thanks.

Actions #13

Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF