Project

General

Profile

Bug #22292

mds: scrub may mark repaired directory with lost dentries and not flush backtrace

Added by Patrick Donnelly about 1 year ago. Updated 8 months ago.

Status:
New
Priority:
High
Category:
fsck/damage handling
Target version:
Start date:
11/30/2017
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, qa-suite
Labels (FS):
qa
Pull request ID:

Description

Simple reproducer (with selected output):

$ ../src/vstart.sh -d -b -l -n
$ # mount at ./mnt
$ cp -a /usr mnt
^C
$ bin/ceph daemon mds.c flush journal
$ bin/rados --pool=cephfs_metadata_a listomapkeys 1000000001c.00000000
site-functions_head
$ bin/rados --pool=cephfs_metadata_a getxattr 1000000001c.00000000 parent
�       zsh]shareWlocaluusr+
$ bin/rados --pool=cephfs_metadata_a getxattr 1000000001d.00000000 parent
�"site-functions        zsh]shareWlocaluusr+
$ bin/rados --pool=cephfs_metadata_a rm 1000000001c.00000000
$ bin/ceph daemon mds.b scrub_path / recursive force repair
$ grep -B2 'scrub complete' < out/mds.b.log 
2017-11-30 18:11:42.798 7fe62a268700 10 log_client  logged 2017-11-30 18:11:37.052016 mds.b mds.0 127.0.0.1:6839/1510709700 1 : cluster [WRN] bad backtrace on inode 0x1000000001c(/usr/local/share/zsh), rewriting it
2017-11-30 18:11:42.798 7fe62a268700 10 log_client  logged 2017-11-30 18:11:37.052050 mds.b mds.0 127.0.0.1:6839/1510709700 2 : cluster [INF] Scrub repaired inode 0x1000000001c (/usr/local/share/zsh)
2017-11-30 18:11:42.798 7fe62a268700 10 log_client  logged 2017-11-30 18:11:37.346361 mds.b mds.0 127.0.0.1:6839/1510709700 3 : cluster [INF] scrub complete
$ bin/rados --pool=cephfs_metadata_a getxattr 1000000001c.00000000 parent
# ???? I thought scrub would flush the backtrace?
$ bin/ceph daemon mds.b flush journal
$ bin/rados --pool=cephfs_metadata_a getxattr 1000000001c.00000000 parent
�       zsh]sharellocal�usr:pdo
$ bin/rados --pool=cephfs_metadata_a listomapkeys 1000000001c.00000000
$

so "site-functions" dentry is lost but the directory is considered repaired.

History

#1 Updated by Patrick Donnelly about 1 year ago

Consensus during scrub is that this can be resolved by adding an appropriate warning to scrub output that the directory inode was recreated and the operator needs to run cephfs-data-scan to recover the lost dentries.

#2 Updated by Patrick Donnelly about 1 year ago

Greg also brought up some good points that we should also mark the directory as damaged (especially in a persistent way that survives fail-over if it doesn't already exist) and/or go read-only so that the operator must resolve the damage before allowing other I/O which could create inconsistency.

#3 Updated by Patrick Donnelly 8 months ago

  • Target version set to v14.0.0
  • Component(FS) qa-suite added
  • Labels (FS) qa added

Also available in: Atom PDF