Project

General

Profile

Bug #19811

rbd-mirror replay fails on attempting to reclaim data to local site (LS) from distant-end after DE promotion.

Added by Eric M Gonzalez 6 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Target version:
-
Start date:
04/28/2017
Due date:
% Done:

0%

Source:
Tags:
rbd-mirror resync split-brain
Backport:
kraken,jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rbd
Release:
Needs Doc:
No

Description

in the event that the primary pool (on the local-site) becomes disconnected and empty without being recreated (say, `rbd rm` all images in the pool) on reconnection of the mirror, under normal operation, the distant-end will notice the now empty pool and then clear itself.

this can be prevented by promoting the images on the distant-end before the local-site pool is brought back on-line. the desire now is to recover the data from the distant-end back to the LS. when the LS comes back on-line and the pool mirroring is established, the LS pool will begin the process of replaying all images marked primary on the distant-end. however, any image that originally belonged to the primary pool in the first place will not complete the mirror process due to a "split-brain" situation; an empty image "placeholder" will be created in the pool but no data will be transferred. this is caused by something in the local-site pool (it was emptied, not re-created) still thinking that those images belong to it and are (in my estimation) still treated as primary.

it is possible to trigger the mirror playback in one of three ways:
1) by issuing `rbd mirror image resync` on the desired images in the LS pool,
2) restarting the `rbd-mirror` on the local-site,
3) if the LS pool is re-created and then re-linked via `rbd-mirror` all images marked primary in the distant-end will be replayed as normal (full copy of all data) on the initial mirror connection.

the issue, then, is that the primary pool does not completely forget about the images when they are deleted. though the above work-arounds are capable of returning the data back to the original primary pool:
- why does the local-site pool still think it owns the data?
- is it the journal in the pool or some other piece of meta-data or image configuration that lies hidden?
- could Ceph / RBD be extended to completely remove this left-over data so that the "emptied" scenario behaves similarly to the "re-created" scenario?"

reproduction:
1) establish a mirror between LS (local-site) and DE (distant-end) with LS being the primary pool.
2) let the mirror do it's initial work replaying LS to DE
3) once the journal replaying is complete stop the rbd-mirror service on DE
4) delete target (or all) images in the LS pool
5) promote target (or all) images in the DE to primary
6) reconnect the two pools by starting rbd-mirror on the LS.

expected result:
- that the LS will fully copy all of the images marked primary in the DE.

actual results:
a) starting rbd-mirror on LS will complain that it cannot open files it thinks that it knows about.
b) "rbd mirror pool status --verbose" for LS will show images reappear as if they were either supposed to be there originally or attempted to be copied from the DE.
c) these images will be empty (all zeroes)

work-arounds:
1) restart the rbd-mirror daemon on LS. this will trigger a resync.
2) manually trigger a resync using "rbd mirror image resync"
3) completely re-creating the LS pool / ceph cluster.

ls.rbd-mirror.issue-19811 - commands run from the LS (as used in description above). runs through "no image", "initial mirror", "deletion and promotion", and the attempt to get it back into LS. (3.45 KB) Eric M Gonzalez, 05/03/2017 05:45 PM


Related issues

Copied to rbd - Backport #20022: kraken: rbd-mirror replay fails on attempting to reclaim data to local site (LS) from distant-end after DE promotion. Resolved
Copied to rbd - Backport #20023: jewel: rbd-mirror replay fails on attempting to reclaim data to local site (LS) from distant-end after DE promotion. Resolved

History

#1 Updated by Jason Dillaman 6 months ago

  • Assignee set to Jason Dillaman

#2 Updated by Jason Dillaman 6 months ago

  • Status changed from New to Need More Info

@Eric: would it be possible for you to retest this using 10.2.7 client software? I used your steps and wasn't able to repeat this issue when running 10.2.7 rbd-mirror.

#3 Updated by Eric M Gonzalez 6 months ago

Sure -- same results.

I've attached a log of operations that you can follow along and see what's happening. I hope that it shows what's going on better than my (too) verbose description above. Let me know if there's anything else I can get for you.

#4 Updated by Jason Dillaman 6 months ago

  • Status changed from Need More Info to In Progress

@Eric: thanks, I see the issue now.

#5 Updated by Jason Dillaman 6 months ago

  • Status changed from In Progress to Need Review
  • Backport set to kraken,jewel
  • Release deleted (jewel)

#6 Updated by Ken Dreyer 6 months ago

  • Status changed from Need Review to Pending Backport

#7 Updated by Nathan Cutler 5 months ago

  • Copied to Backport #20022: kraken: rbd-mirror replay fails on attempting to reclaim data to local site (LS) from distant-end after DE promotion. added

#8 Updated by Nathan Cutler 5 months ago

  • Copied to Backport #20023: jewel: rbd-mirror replay fails on attempting to reclaim data to local site (LS) from distant-end after DE promotion. added

#9 Updated by Nathan Cutler 5 months ago

@Jason The test in the master commit does not backport cleanly to jewel. It appears to depend on 19dd5a82bb8 but the test in that one doesn't backport cleanly, either. Can you suggest how to proceed?

#10 Updated by Jason Dillaman 5 months ago

@Nathan: feel free to re-assign this one to me and I'll make the necessary changes.

#11 Updated by Nathan Cutler 5 months ago

OK, #20023 reassigned to Jason.

#12 Updated by Nathan Cutler about 2 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF