Bug #23516
[rbd-mirror] entries_behind_master will not be zero after mirror over
0%
Description
I have two ceph-12.2.4 cluster. Rbd mirror deamon run on cluster two. poolclz/rbdclz on cluster one is primary image and non-primary on cluster two.
When I write data in to poolclz/rbdclz by using fio or rbd bench, rbd mirror will data to cluster.
I find the entries_behind_master can't decrease to zero by using command "rbd mirror image status poolclz/rbdclz". That mean mirror_position is some entries behind master_position.I think this is unacceptable.
rbdclz:
global_id: 105d7a59-06c0-404d-878e-80dadba39c55
state: up+replaying
description: replaying, master_position=[object_number=7, tag_tid=30, entry_tid=295], mirror_position=[object_number=7, tag_tid=30, entry_tid=287], entries_behind_master=8
last_update: 2018-03-23 20:09:39
Related issues
History
#1 Updated by Jason Dillaman almost 6 years ago
- Priority changed from Normal to Low
This is actually just an artifact of IO batching within rbd-mirror. The commit position is only updated after every 32 IOs in the absence of a flush request. Under a real workload, you would have flush requests so the updated commit position would be appropriately advanced. If you were to take a snapshot of your source image and compare it to your destination, you would discover that both images are identical (you would also discover your 'entries_behind_master' dropping to zero since a snapsoht would flush).
#2 Updated by liuzhong chen almost 6 years ago
Thank you! @Jason Dillaman
I have anther two question.I find the non-primary image used volumes is equal to primary image used volumes although the entries_behind_master is not zero.It means just the mirror_commit_position not update but all the date have been mirror to mirror image.Am I right?
[root@node-1 ~]# rbd du poolclz1/rbdclz1
NAME PROVISIONED USED
rbdclz1 1024M 940M
[root@node-1 ~]# ssh node-2
Last login: Fri Mar 30 01:51:34 2018 from 192.168.122.1
[root@node-2 ~]# rbd mirror image status poolclz1/rbdclz1
rbdclz1:
global_id: 09febb93-3eac-4467-a284-f02f90b846b5
state: up+replaying
description: replaying, master_position=[object_number=3, tag_tid=2, entry_tid=619], mirror_position=[object_number=3, tag_tid=2, entry_tid=607], entries_behind_master=12
last_update: 2018-03-30 02:08:19
[root@node-2 ~]# rbd du poolclz1/rbdclz1
NAME PROVISIONED USED
rbdclz1 1024M 940M
#3 Updated by liuzhong chen almost 6 years ago
Another question is if I use fio to write for 60s.Is there any other parameter to indicate that the mirror is over?
If there is no other paramete,I think if the entries_behind_master is less than 32,I can affirm the mirror is over.Whether this method is feasible or not.
Thank you!
#4 Updated by Jason Dillaman almost 6 years ago
We'll get this status artifact fixed. You can use the admin socket to the rbd-mirror daemon to force a flush if desired: 'ceph --admin-daemon /path/to/rbd-mirror.asok rbd mirror flush'.
#5 Updated by liuzhong chen almost 6 years ago
Thank you! This method can flush it.But now need to write a script to judge the mirror is over automatic,so I wonder if it is all data of primary image have been mirrored to non-primary although the entries_behind_master stop at 12?
#6 Updated by Jason Dillaman almost 6 years ago
- Subject changed from rbd mirror: entries_behind_master will not be zero after mirror over to [rbd-mirror] entries_behind_master will not be zero after mirror over
- Status changed from New to In Progress
- Assignee set to Jason Dillaman
- Backport set to jewel,luminous,mimic
#7 Updated by Jason Dillaman almost 6 years ago
- Status changed from In Progress to Fix Under Review
#8 Updated by Jason Dillaman almost 6 years ago
- Status changed from Fix Under Review to Pending Backport
#9 Updated by Jason Dillaman almost 6 years ago
- Copied to Backport #24378: luminous: [rbd-mirror] entries_behind_master will not be zero after mirror over added
#10 Updated by Jason Dillaman almost 6 years ago
- Status changed from Pending Backport to Fix Under Review
#11 Updated by Mykola Golub almost 6 years ago
- Status changed from Fix Under Review to Pending Backport
#12 Updated by Nathan Cutler almost 6 years ago
- Copied to Backport #24391: mimic: [rbd-mirror] entries_behind_master will not be zero after mirror over added
#13 Updated by Nathan Cutler almost 6 years ago
- Copied to Backport #24392: jewel: [rbd-mirror] entries_behind_master will not be zero after mirror over added
#14 Updated by Jason Dillaman over 5 years ago
- Status changed from Pending Backport to Resolved