Project

General

Profile

Actions

Bug #24045

closed

Eviction still raced with scrub due to preemption

Added by David Zafman almost 6 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
David Zafman
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We put code in cache tier eviction to check the scrub range, but that isn't sufficient. During scrub preemption restart we are selecting a smaller chunk. This allows eviction to evict something that had already been sent to the replica. The cleaned_meta_map can buffer the ScrubMap for a new evicted snapshot. The eviction will be received by the replica and the head object will be modified before _scan_snaps() is called with the removed snapshot still present.

snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired

http://pulpito.ceph.com/kchai-2018-05-05_14:56:43-rados-wip-kefu-testing-2018-05-05-1912-distro-basic-smithi/

$ git br --contains 83861a5b75ddb98366f1ec106487b88703f25cf7 | grep wip-kefu-testing-2018-05-05-1912
  • wip-kefu-testing-2018-05-05-1912

Piecing together the sequence of events:

Chunk including smithi16216580-301...:2e but NOT its head object sent to replica
Replica stores smithi16216580-301...:2e in cleaned_meta_map
Preemption causes new chunk which does NOT include smithi16216580-301...:2e
Eviction is able to delete smithi16216580-301...:2e from cache tier
Replica sees delete and removes snapshot from head object
Scrub gets back to handling a chunk which includes smithi16216580-301...:head
Replica _scan_snaps() outputs error because snapshot 2e is present but head doesn't include it.


Related issues 5 (0 open5 closed)

Related to RADOS - Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repairedResolvedDavid Zafman04/27/2018

Actions
Related to RADOS - Bug #22881: scrub interaction with HEAD boundaries and snapmapper repair is brokenResolvedDavid Zafman02/01/2018

Actions
Related to RADOS - Bug #38840: snaps missing in mapper, should be: ca was r -2...repairedResolvedDavid Zafman

Actions
Copied to RADOS - Backport #24149: mimic: Eviction still raced with scrub due to preemptionResolvedDavid ZafmanActions
Copied to RADOS - Backport #24153: luminous: Eviction still raced with scrub due to preemptionResolvedDavid ZafmanActions
Actions #1

Updated by David Zafman almost 6 years ago

  • Priority changed from Normal to Urgent
Actions #2

Updated by David Zafman almost 6 years ago

  • Related to Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired added
Actions #3

Updated by David Zafman almost 6 years ago

  • Related to Bug #22881: scrub interaction with HEAD boundaries and snapmapper repair is broken added
Actions #4

Updated by Sage Weil almost 6 years ago

  • Status changed from 12 to Pending Backport
  • Backport changed from luminous to luminous,mimic
Actions #5

Updated by David Zafman almost 6 years ago

  • Copied to Backport #24149: mimic: Eviction still raced with scrub due to preemption added
Actions #6

Updated by David Zafman almost 6 years ago

  • Copied to Backport #24153: luminous: Eviction still raced with scrub due to preemption added
Actions #7

Updated by David Zafman almost 6 years ago

  • Status changed from Pending Backport to Resolved
Actions #8

Updated by David Zafman about 5 years ago

  • Pull request ID set to 21892
Actions #9

Updated by David Zafman about 5 years ago

  • Related to Bug #38840: snaps missing in mapper, should be: ca was r -2...repaired added
Actions

Also available in: Atom PDF