Project

General

Profile

Bug #23516

[rbd-mirror] entries_behind_master will not be zero after mirror over

Added by liuzhong chen 10 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Low
Target version:
Start date:
03/30/2018
Due date:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
jewel,luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rbd
Pull request ID:

Description

I have two ceph-12.2.4 cluster. Rbd mirror deamon run on cluster two. poolclz/rbdclz on cluster one is primary image and non-primary on cluster two.
When I write data in to poolclz/rbdclz by using fio or rbd bench, rbd mirror will data to cluster.
I find the entries_behind_master can't decrease to zero by using command "rbd mirror image status poolclz/rbdclz". That mean mirror_position is some entries behind master_position.I think this is unacceptable.

rbdclz:
  global_id:   105d7a59-06c0-404d-878e-80dadba39c55
  state:       up+replaying
  description: replaying, master_position=[object_number=7, tag_tid=30, entry_tid=295], mirror_position=[object_number=7, tag_tid=30, entry_tid=287], entries_behind_master=8
  last_update: 2018-03-23 20:09:39


Related issues

Copied to rbd - Backport #24378: luminous: [rbd-mirror] entries_behind_master will not be zero after mirror over Resolved
Copied to rbd - Backport #24391: mimic: [rbd-mirror] entries_behind_master will not be zero after mirror over Resolved
Copied to rbd - Backport #24392: jewel: [rbd-mirror] entries_behind_master will not be zero after mirror over Closed

History

#1 Updated by Jason Dillaman 10 months ago

  • Priority changed from Normal to Low

This is actually just an artifact of IO batching within rbd-mirror. The commit position is only updated after every 32 IOs in the absence of a flush request. Under a real workload, you would have flush requests so the updated commit position would be appropriately advanced. If you were to take a snapshot of your source image and compare it to your destination, you would discover that both images are identical (you would also discover your 'entries_behind_master' dropping to zero since a snapsoht would flush).

#2 Updated by liuzhong chen 10 months ago

Thank you! @Jason Dillaman
I have anther two question.I find the non-primary image used volumes is equal to primary image used volumes although the entries_behind_master is not zero.It means just the mirror_commit_position not update but all the date have been mirror to mirror image.Am I right?

[root@node-1 ~]# rbd du poolclz1/rbdclz1
NAME    PROVISIONED USED
rbdclz1       1024M 940M

[root@node-1 ~]# ssh node-2
Last login: Fri Mar 30 01:51:34 2018 from 192.168.122.1
[root@node-2 ~]# rbd mirror image status poolclz1/rbdclz1
rbdclz1:
  global_id:   09febb93-3eac-4467-a284-f02f90b846b5
  state:       up+replaying
  description: replaying, master_position=[object_number=3, tag_tid=2, entry_tid=619], mirror_position=[object_number=3, tag_tid=2, entry_tid=607], entries_behind_master=12
  last_update: 2018-03-30 02:08:19

[root@node-2 ~]# rbd du poolclz1/rbdclz1
NAME    PROVISIONED USED
rbdclz1       1024M 940M

#3 Updated by liuzhong chen 10 months ago

Another question is if I use fio to write for 60s.Is there any other parameter to indicate that the mirror is over?
If there is no other paramete,I think if the entries_behind_master is less than 32,I can affirm the mirror is over.Whether this method is feasible or not.
Thank you!

#4 Updated by Jason Dillaman 10 months ago

We'll get this status artifact fixed. You can use the admin socket to the rbd-mirror daemon to force a flush if desired: 'ceph --admin-daemon /path/to/rbd-mirror.asok rbd mirror flush'.

#5 Updated by liuzhong chen 10 months ago

Thank you! This method can flush it.But now need to write a script to judge the mirror is over automatic,so I wonder if it is all data of primary image have been mirrored to non-primary although the entries_behind_master stop at 12?

#6 Updated by Jason Dillaman 8 months ago

  • Subject changed from rbd mirror: entries_behind_master will not be zero after mirror over to [rbd-mirror] entries_behind_master will not be zero after mirror over
  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman
  • Backport set to jewel,luminous,mimic

#7 Updated by Jason Dillaman 8 months ago

  • Status changed from In Progress to Need Review

#8 Updated by Jason Dillaman 8 months ago

  • Status changed from Need Review to Pending Backport

#9 Updated by Jason Dillaman 8 months ago

  • Copied to Backport #24378: luminous: [rbd-mirror] entries_behind_master will not be zero after mirror over added

#10 Updated by Jason Dillaman 8 months ago

  • Status changed from Pending Backport to Need Review

#11 Updated by Mykola Golub 8 months ago

  • Status changed from Need Review to Pending Backport

#12 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #24391: mimic: [rbd-mirror] entries_behind_master will not be zero after mirror over added

#13 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #24392: jewel: [rbd-mirror] entries_behind_master will not be zero after mirror over added

#14 Updated by Jason Dillaman 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF