Project

General

Profile

Bug #53914

rbd mirror snapshot schedule is not working properly after a few failover-relocate operations

Added by Sunny Kumar about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus,pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Steps to reproduce:
[NOTE: this is not a consistent reproducer]
- Enable mirroring and ensure RBD mirror daemons are running
- Demote image on primary cluster
- Promote image on the Secondary cluster
- Do IO on Secondary
- Sleep for 75 seconds (considering schedule is for 60 sec)
- Verify snapshots are being created for added schedule
- Demote on Secondary
- Promote on Primary
- Sleep for 75 seconds (considering schedule is for 60 sec)
- Verify snapshots are being created for added schedule

Sometimes snapshots are not created by scheduler.


Related issues

Copied to rbd - Backport #56144: pacific: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations Resolved
Copied to rbd - Backport #56145: octopus: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations Resolved
Copied to rbd - Backport #56146: quincy: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations Resolved

History

#1 Updated by Sunny Kumar about 1 year ago

  • Status changed from New to In Progress
  • Assignee set to Sunny Kumar

#2 Updated by Sunny Kumar about 1 year ago

$ rbd mirror snapshot schedule status

SCHEDULE TIME IMAGE
2022-01-20 16:45:00 replicapool/test-2
2022-01-20 16:45:00 replicapool/test-3

$ rbd mirror snapshot schedule ls -R
POOL NAMESPACE IMAGE SCHEDULE
replicapool test-1 every 1m **
replicapool test-2 every 1m
replicapool test-3 every 1m

Schedule is present in schedule list but somehow its not getting added to scheduler queue.

#3 Updated by Sunny Kumar about 1 year ago

There are 2 queues we maintain for snapshot scheduling; for creating the next snapshot and another mirror snapshot schedule. Both of them are reloaded every 60 Sec.I see when we fetch from the mirror snapshot queue it does not show newly added schedules for problematic images.So, when we add a new schedule at the next load it should refresh images and add a newly added schedule to the mirror schedule queue. I suspect it's not happening may be because mgr-watchers are unregistered or we are missing some corner cases.

#4 Updated by Ilya Dryomov 8 months ago

  • Assignee changed from Sunny Kumar to Ilya Dryomov

#5 Updated by Ilya Dryomov 8 months ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to octopus,pacific,quincy
  • Pull request ID set to 46743

#6 Updated by Ilya Dryomov 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#7 Updated by Backport Bot 7 months ago

  • Copied to Backport #56144: pacific: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations added

#8 Updated by Backport Bot 7 months ago

  • Copied to Backport #56145: octopus: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations added

#9 Updated by Backport Bot 7 months ago

  • Copied to Backport #56146: quincy: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations added

#10 Updated by Ilya Dryomov 6 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF