Project

General

Profile

Bug #16691

sepia LRC lost directories

Added by Greg Farnum about 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
fsck/damage handling
Target version:
-
Start date:
07/14/2016
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

If you log in to the sepia long-running cluster, it has 37 directories whose objects it lost.

I spot-checked one of them, and the object exists but has zero size and no omap keys. All the directories are old (early 2014 at the newest) and they appear to be sequentially-numbered. This sum of evidence makes me think that we ran the tmap2omap upgrade tool on it and something went wrong with these older directories (perhaps older encodings weren't handled correctly).

History

#1 Updated by Zheng Yan about 3 years ago

what do you mean they are old? what does 'rados stat xxxx' show?

#2 Updated by John Spray about 3 years ago

  • Assignee set to Greg Farnum

#3 Updated by John Spray about 3 years ago

Plan is for greg to look into the TMAP2OMAP OSD code to look for what might have causd that.

Afterwards John+Doug will get into trying to clean up the cluster with our repair tools.

#4 Updated by Greg Farnum about 3 years ago

  • Subject changed from sepia LRC lost directories (tmap2omap went bad?) to sepia LRC lost directories
  • Category changed from Correctness/Safety to fsck/damage handling
  • Assignee changed from Greg Farnum to John Spray

Well, I checked the code again and the tmap2omap path looks appropriately durable.

I did notice one thing that helps explain it a little: we pass a "nullok" flag when invoking tmap2omap, which makes the operation succeed even if there is no data present in the object. This is required, since if the directory is already an omap, there is no tmap data. But it means a previously-broken object won't get detected during this upgrade. :(

Anyway, this cluster has been damaged in various ways in the past. I think these directories simply got broken in the depths of time and are only now being noticed (so, hurray damage detection!).

Assigning to John for him and Doug to clean up.

#5 Updated by John Spray about 3 years ago

(Mainly for my reference) etherpad from repairing is here http://etherpad.corp.redhat.com/efev9SA7rn

#6 Updated by John Spray about 3 years ago

  • Priority changed from Urgent to High

The offending dentries that point to damaged dirfrags have been removed (by removing the omap keys). The objects themselves are still in the system but not in a way that will make it unhappy.

#7 Updated by John Spray over 2 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF