Ceph » CephFS

Venky Shankar wrote:

Tracker https://tracker.ceph.com/issues/54546 related to metadata corruption which is seen when running databases (esp. PostgreSQL) on Ceph File System. While we work on identifying the bug, we need to provide users a way to infer such corruption which might exists in the cluster (possible, if they ran PostgreSQL on CephFS).

The tool should scan the inodes in the metadata pool and alert for any anomalies in file metadata (and optionally fix it, which is a bit tricky atm), E.g.: the corruption mentioned above clobbers an inodes' "first" field (to -2,head or to a value which is not sane). Maybe, we would want to include these checks as part of file system scrub.

Normally metadata pool scans is achieved via MDS scrub. You are suggesting a separate tool?

Patrick Donnelly wrote:

Venky Shankar wrote:

Tracker https://tracker.ceph.com/issues/54546 related to metadata corruption which is seen when running databases (esp. PostgreSQL) on Ceph File System. While we work on identifying the bug, we need to provide users a way to infer such corruption which might exists in the cluster (possible, if they ran PostgreSQL on CephFS).

The tool should scan the inodes in the metadata pool and alert for any anomalies in file metadata (and optionally fix it, which is a bit tricky atm), E.g.: the corruption mentioned above clobbers an inodes' "first" field (to -2,head or to a value which is not sane). Maybe, we would want to include these checks as part of file system scrub.

Normally metadata pool scans is achieved via MDS scrub. You are suggesting a separate tool?

I'm fine with having this as part of scrub.

Posting an update here based on discussion between me, Greg and Patrick:

Short term plan: Helper script to identify corrupted files - this will be an offline script (does not require ceph-mds to be available) that would list out corrupted files. The script will not perform any remedial action on the corrupted files though. The idea is to provide the user with a tool to assess the extent of corruption in the file system and perform any backup operation if required (such as copying the file out from the file system, etc..). This tool will be maintained in the ceph repo.

Long term plan: Fix the bug (obviously!), but, at the same time assess if its really necessary to assert when certain conditions are not satisfied (such as this one in the unlink path).

Ken Dreyer wrote:

https://github.com/ceph/ceph/pull/47542 will be in reef, and we need this in quincy.

The reason this wasn't a candidate for backport is that the tool was very much standalone and not packaged. Users using the tool directly fetched it from the ceph repo (although care is taken to have the tool work with older ceph versions - p, q and now r).