sepia LRC lost directories
If you log in to the sepia long-running cluster, it has 37 directories whose objects it lost.
I spot-checked one of them, and the object exists but has zero size and no omap keys. All the directories are old (early 2014 at the newest) and they appear to be sequentially-numbered. This sum of evidence makes me think that we ran the tmap2omap upgrade tool on it and something went wrong with these older directories (perhaps older encodings weren't handled correctly).
#4 Updated by Greg Farnum about 3 years ago
- Subject changed from sepia LRC lost directories (tmap2omap went bad?) to sepia LRC lost directories
- Category changed from Correctness/Safety to fsck/damage handling
- Assignee changed from Greg Farnum to John Spray
Well, I checked the code again and the tmap2omap path looks appropriately durable.
I did notice one thing that helps explain it a little: we pass a "nullok" flag when invoking tmap2omap, which makes the operation succeed even if there is no data present in the object. This is required, since if the directory is already an omap, there is no tmap data. But it means a previously-broken object won't get detected during this upgrade. :(
Anyway, this cluster has been damaged in various ways in the past. I think these directories simply got broken in the depths of time and are only now being noticed (so, hurray damage detection!).
Assigning to John for him and Doug to clean up.