Bug #53914: rbd mirror snapshot schedule is not working properly after a few failover-relocate operations - rbd - Ceph

Actions

Copy link

Bug #53914

closed

rbd mirror snapshot schedule is not working properly after a few failover-relocate operations

Added by Sunny Kumar over 2 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Ilya Dryomov

Target version:

% Done:

Source:

Tags:

Backport:

octopus,pacific,quincy

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

46743

Crash signature (v1):

Crash signature (v2):

Description

Steps to reproduce:
[NOTE: this is not a consistent reproducer]
- Enable mirroring and ensure RBD mirror daemons are running
- Demote image on primary cluster
- Promote image on the Secondary cluster
- Do IO on Secondary
- Sleep for 75 seconds (considering schedule is for 60 sec)
- Verify snapshots are being created for added schedule
- Demote on Secondary
- Promote on Primary
- Sleep for 75 seconds (considering schedule is for 60 sec)
- Verify snapshots are being created for added schedule

Sometimes snapshots are not created by scheduler.

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Sunny Kumar over 2 years ago

Status changed from New to In Progress
Assignee set to Sunny Kumar

Actions

Copy link

Updated by Sunny Kumar over 2 years ago

$ rbd mirror snapshot schedule status

SCHEDULE TIME IMAGE
2022-01-20 16:45:00 replicapool/test-2
2022-01-20 16:45:00 replicapool/test-3

$ rbd mirror snapshot schedule ls -R
POOL NAMESPACE IMAGE SCHEDULE
replicapool test-1 every 1m **
replicapool test-2 every 1m
replicapool test-3 every 1m

Schedule is present in schedule list but somehow its not getting added to scheduler queue.

Actions

Copy link

Updated by Sunny Kumar over 2 years ago

There are 2 queues we maintain for snapshot scheduling; for creating the next snapshot and another mirror snapshot schedule. Both of them are reloaded every 60 Sec.I see when we fetch from the mirror snapshot queue it does not show newly added schedules for problematic images.So, when we add a new schedule at the next load it should refresh images and add a newly added schedule to the mirror schedule queue. I suspect it's not happening may be because mgr-watchers are unregistered or we are missing some corner cases.

Actions

Copy link