Project

General

Profile

Bug #44159

[rbd-mirror] Mirror daemon never recovers from being blacklisted

Added by Oliver Freyermuth 7 months ago. Updated 7 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous,mimic,nautilus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

I can reproduce this rather reliably by:
- Restarting many OSDs (old nodes, slow spinning disks, likely exceeding default blacklist timeout).
- Sometimes, it also happens when restarting other RBD mirror daemons (we have 3).

The attached log is extracted from one blacklisted RBD mirror unable to recover at log level 15.
RBD volume names and domains are sanitized, otherwise the log is untouched.

ceph-client.rbd_mirror_backup.log.gz (479 KB) Oliver Freyermuth, 02/15/2020 05:18 PM


Related issues

Copied to rbd - Backport #44262: mimic: [rbd-mirror] Mirror daemon never recovers from being blacklisted Resolved
Copied to rbd - Backport #44263: nautilus: [rbd-mirror] Mirror daemon never recovers from being blacklisted Resolved
Copied to rbd - Backport #44264: luminous: [rbd-mirror] Mirror daemon never recovers from being blacklisted In Progress

History

#1 Updated by Mykola Golub 7 months ago

  • Status changed from New to In Progress
  • Assignee set to Mykola Golub

#2 Updated by Mykola Golub 7 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 33411

In the provided log there are many messages like these ones:

2020-02-14 02:14:56.653 7f42f7ac1700 -1 rbd::mirror::InstanceReplayer: 0x55bab1dc3b80 start_image_replayer: global_image_id=446b538f-1f61-4daa-b05f-93f76cd5e652: blacklisted detected during image replay

2020-02-14 02:14:56.660 7f42f7ac1700  5 rbd::mirror::LeaderWatcher: 0x55bab29a9200 handle_rewatch_complete: r=-108

So both rbd-mirror's InstanceReplayer and LeaderWatcher detected the "blacklisted" state but it was not propagated on the higher level to restart the PoolReplayer.

#3 Updated by Jason Dillaman 7 months ago

  • Backport set to luminous,mimic,nautilus

#4 Updated by Jason Dillaman 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#5 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #44262: mimic: [rbd-mirror] Mirror daemon never recovers from being blacklisted added

#6 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #44263: nautilus: [rbd-mirror] Mirror daemon never recovers from being blacklisted added

#7 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #44264: luminous: [rbd-mirror] Mirror daemon never recovers from being blacklisted added

Also available in: Atom PDF