Project

General

Profile

Bug #24045

Eviction still raced with scrub due to preemption

Added by David Zafman over 1 year ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
05/07/2018
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

We put code in cache tier eviction to check the scrub range, but that isn't sufficient. During scrub preemption restart we are selecting a smaller chunk. This allows eviction to evict something that had already been sent to the replica. The cleaned_meta_map can buffer the ScrubMap for a new evicted snapshot. The eviction will be received by the replica and the head object will be modified before _scan_snaps() is called with the removed snapshot still present.

snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired

http://pulpito.ceph.com/kchai-2018-05-05_14:56:43-rados-wip-kefu-testing-2018-05-05-1912-distro-basic-smithi/

$ git br --contains 83861a5b75ddb98366f1ec106487b88703f25cf7 | grep wip-kefu-testing-2018-05-05-1912
  • wip-kefu-testing-2018-05-05-1912

Piecing together the sequence of events:

Chunk including smithi16216580-301...:2e but NOT its head object sent to replica
Replica stores smithi16216580-301...:2e in cleaned_meta_map
Preemption causes new chunk which does NOT include smithi16216580-301...:2e
Eviction is able to delete smithi16216580-301...:2e from cache tier
Replica sees delete and removes snapshot from head object
Scrub gets back to handling a chunk which includes smithi16216580-301...:head
Replica _scan_snaps() outputs error because snapshot 2e is present but head doesn't include it.


Related issues

Related to RADOS - Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired Resolved 04/27/2018
Related to RADOS - Bug #22881: scrub interaction with HEAD boundaries and snapmapper repair is broken Resolved 02/01/2018
Related to RADOS - Bug #38840: snaps missing in mapper, should be: ca was r -2...repaired Pending Backport 03/21/2019
Copied to RADOS - Backport #24149: mimic: Eviction still raced with scrub due to preemption Resolved
Copied to RADOS - Backport #24153: luminous: Eviction still raced with scrub due to preemption Resolved

History

#1 Updated by David Zafman over 1 year ago

  • Priority changed from Normal to Urgent

#2 Updated by David Zafman over 1 year ago

  • Related to Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired added

#3 Updated by David Zafman over 1 year ago

  • Related to Bug #22881: scrub interaction with HEAD boundaries and snapmapper repair is broken added

#4 Updated by Sage Weil over 1 year ago

  • Status changed from Verified to Pending Backport
  • Backport changed from luminous to luminous,mimic

#5 Updated by David Zafman over 1 year ago

  • Copied to Backport #24149: mimic: Eviction still raced with scrub due to preemption added

#6 Updated by David Zafman over 1 year ago

  • Copied to Backport #24153: luminous: Eviction still raced with scrub due to preemption added

#7 Updated by David Zafman over 1 year ago

  • Status changed from Pending Backport to Resolved

#8 Updated by David Zafman 7 months ago

  • Pull request ID set to 21892

#9 Updated by David Zafman 7 months ago

  • Related to Bug #38840: snaps missing in mapper, should be: ca was r -2...repaired added

Also available in: Atom PDF