Ceph » CephFS

Hello developers,

A customer has a cluster which currently has 4 large OMAP objects (one old and three new) in its metadata pool. I am aware of https://tracker.ceph.com/issues/45333, and in this comment https://tracker.ceph.com/issues/45333#note-6 a procedure for triggering the directory fragmentation exists: reconstruct the directory path and list that directory to get it fragmented. However, in our case, this procedure is inapplicable.

# rados getxattr --pool=mainfs.meta 100290d9cb3.00000000 parent | ceph-dencoder type inode_backtrace_t import - decode dump_json
{
    "ino": 1100200385715,
    "ancestors": [
        {
            "dirino": 1543,
            "dname": "100290d9cb3",
            "version": 318702055
        },
        {
            "dirino": 256,
            "dname": "stray7",
            "version": 1405762425
        }
    ],
    "pool": 2,
    "old_pools": []
}

See - it is a stray. Actually, all three new large OMAP objects correspond to stray directories that for this reason cannot be listed. Instructions should be provided on how to deal with this situation.

Regarding possible snapshots: the oldest snapshot of a directory that "officially" should have snapshots is dated January 28, 2024. There might be older snapshots of other directories, I have not searched for them and I don't know if they exist.

Regarding the contents of one of the stray objects, I did this to get some statistics:

# ceph tell mds.0 dump tree "~mdsdir/stray7" > stray7.json
# ls -l stray7.json 
-rw-r--r-- 1 root root 710084873 Feb  2 08:25 stray7.json
# wc -l stray7.json 
23391176 stray7.json
# grep stray_prior_path stray7.json | wc -l
135172
# grep stray_prior_path stray7.json | grep -v '"stray_prior_path": ""' | wc -l
358

I can confirm that the entries with non-empty stray_prior_path are "clustered" in two different directories. I have checked one entry manually - it does not exist as either a file or a directory, but its parent does and contains a lot of existing subdirectories named in a similar way.