Ceph » CephFS

After carefull consideration, i try to fix this bug using first solution.
function inject_with_backtrace will traverse a ojbect's ancestor
when we inject_linkage one dentry, we should increate parent dir's fnode.fragstat.nfiles if we didn't create parent directory's omap_header first cycle.If not first cycle we should increase parent dir's fnode.fragstat'nsbudirs after inject_linkage.In order to all hierarchical diretores have most actual fragstat we should traverse all ancestor of the object, do you tink so?

This inconsistent state is created by cephfs-data-scan? I think we can avoid injecting dentry to stray directory (inject it into lost+found instead). MetadataDriver::inject_with_backtrace() calls MetadataDriver::get_frag_of(), MetadataDriver::get_frag_of() loads the directory inode. it's easy to check if the inode is stray there.

Ivan Guan wrote:

After carefull consideration, i try to fix this bug using first solution.
function inject_with_backtrace will traverse a ojbect's ancestor
when we inject_linkage one dentry, we should increate parent dir's fnode.fragstat.nfiles if we didn't create parent directory's omap_header first cycle.If not first cycle we should increase parent dir's fnode.fragstat'nsbudirs after inject_linkage.In order to all hierarchical diretores have most actual fragstat we should traverse all ancestor of the object, do you tink so?

My solution don't change if dentry shuold be injected to stray diretroy or lost+found.I only want update parent directories omap_headr after we function inject_linkage.If we inject a file crosponddingly nfiles should be increased but if inject a diretory we shuold increase the nsbudirs.After my test,do this can ensure all hierarchical diretories have right statistics of nfils and nsbudirs.

Ivan Guan wrote:

Ivan Guan wrote:

After carefull consideration, i try to fix this bug using first solution.
function inject_with_backtrace will traverse a ojbect's ancestor
when we inject_linkage one dentry, we should increate parent dir's fnode.fragstat.nfiles if we didn't create parent directory's omap_header first cycle.If not first cycle we should increase parent dir's fnode.fragstat'nsbudirs after inject_linkage.In order to all hierarchical diretores have most actual fragstat we should traverse all ancestor of the object, do you tink so?

My solution don't change if dentry shuold be injected to stray diretroy or lost+found.I only want update parent directories omap_headr after we function inject_linkage.If we inject a file crosponddingly nfiles should be increased but if inject a diretory we shuold increase the nsbudirs.After my test,do this can ensure all hierarchical diretories have right statistics of nfils and nsbudirs.

This is better, but it still can't ensure fragstats/rstats are correct. Nothing can be 100% trusted during fs recovery. Running "ceph daemon mds.a scrub_path / force recursive repair" can fix wrong fragstat/rstat.

I fixed a cephfs-data-scan bug. https://github.com/ceph/ceph/pull/16202. It can explain this inconsistent state.

Thank you,i made a test of running "ceph daemon mds.a scrub_path / force recursive repair" after scan_inodes.As you said all hierarchical directories have the right fragstat/rstats,but another i find the other serious problem inotable's free and projected_free is {
"0": {
"data": {
"version": 1,
"inotable": {
"projected_free": [ {
"start": 1099511627776,
"len": 1099511627776
}
],
"free": [ {
"start": 1099511627776,
"len": 1099511627776
}
]
}
},
"result": 0
}
}

we i touch a file under mointpoint mds core
2017-07-07 15:45:40.327538 7f7fa12f7700 20 mds.0.sessionmap mark_projected s=0x7f7fb1ff4a80 name=client.18705 pv=4 > 5
2017-07-07 15:45:40.327540 7f7fa12f7700 10 mds.0.server prepare_new_inode prealloc [10000000001~3e8]
2017-07-07 15:45:40.327542 7f7fa12f7700 10 mds.0.server dir mode 041777 new mode 0100644
2017-07-07 15:45:40.330619 7f7fa12f7700 -1 mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)' thread 7f7fa12f7700 time 2017-07-07 15:45:40.327548
mds/MDCache.cc: 265: FAILED assert(inode_map.count(in>vino()) == 0)

ceph version 10.2.2-37-g75b77cc (75b77cca7e6b7f3187518acded5941ffe8f33272)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f7fa7733ee5]
 2: (MDCache::add_inode(CInode*)+0x246) [0x7f7fa73fab76]
 3: (Server::prepare_new_inode(std::shared_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t*)+0x1111) [0x7f7fa7388601]
 4: (Server::handle_client_openc(std::shared_ptr<MDRequestImpl>&)+0xe32) [0x7f7fa73b0a02]
 5: (Server::dispatch_client_request(std::shared_ptr<MDRequestImpl>&)+0xa88) [0x7f7fa73be0d8]
 6: (Server::handle_client_request(MClientRequest*)+0x48c) [0x7f7fa73be69c]
 7: (Server::dispatch(Message*)+0x3eb) [0x7f7fa73c2b0b]
 8: (MDSRank::handle_deferrable_message(Message*)+0x82c) [0x7f7fa733d02c]
 9: (MDSRank::_dispatch(Message*, bool)+0x207) [0x7f7fa7346807]
 10: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x7f7fa7347995]
 11: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x7f7fa732dd03]
 12: (DispatchQueue::entry()+0x78a) [0x7f7fa78336aa]
 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7fa7718e2d]
 14: (()+0x7dc5) [0x7f7fa650ddc5]
 15: (clone()+0x6d) [0x7f7fa4fda28d]

Zheng Yan wrote:

I fixed a cephfs-data-scan bug. https://github.com/ceph/ceph/pull/16202. It can explain this inconsistent state.

For 'cephfs-journal-tool event recover_dentries summary' i think it only can replay the entry which still not be trimmed.In test_data_scan.py we will run
'cephfs-journal-tool journal reset --force',which will clean all the journal.So recover_dentries can't paly any role,what do you think about it?

you can use cephfs-table-tool manually remove used inodes from inotable. (use "rados -p data ls", "rados -p metadata ls" to find the largest used ino number)

I'm writing a patch the for cephfs-data-scan, make it update inotable automatically

Zheng Yan wrote:

you can use cephfs-table-tool manually remove used inodes from inotable. (use "rados -p data ls", "rados -p metadata ls" to find the largest used ino number)

I'm writing a patch the for cephfs-data-scan, make it update inotable automatically

That would be great which will effectively prevent ino reuse.however,i just found out that the "ceph daemon mds.a scrub_path / force recursive repair" can also repaire inotable.Unfortunately,there is a
'EResetJournal' evnet generated by 'cephfs-journal-tool --debug-mds=20 --debug-monc=1 --debug-objecter=20 --debug-rados=20 --cluster xtao journal reset --force' which
make mds skip 989680 inos in test-data-scan.py.All my inos less then 10000989680,so them have not removed from free we excute CInode::validate_disk_state.

mds.log:
2017-07-07 17:13:50.456023 7f014b647700 1 mds.0.journaler(ro) recover start
2017-07-07 17:13:50.456033 7f014b647700 1 mds.0.journaler(ro) read_head
2017-07-07 17:13:50.456208 7f014b647700 4 mds.0.log Waiting for journal 200 to recover...
...
2017-07-07 17:13:50.477537 7f0149f3e700 10 mds.0.log _replay 4194304~28 / 4194352 0.000000: EResetJournal
2017-07-07 17:13:50.477550 7f0149f3e700 1 mds.0.journal EResetJournal
2017-07-07 17:13:50.477557 7f0149f3e700 1 mds.0.sessionmap wipe start
2017-07-07 17:13:50.477561 7f0149f3e700 10 mds.0.sessionmap dump
2017-07-07 17:13:50.477564 7f0149f3e700 1 mds.0.sessionmap wipe result
2017-07-07 17:13:50.477566 7f0149f3e700 10 mds.0.sessionmap dump
2017-07-07 17:13:50.477568 7f0149f3e700 1 mds.0.sessionmap wipe done
2017-07-07 17:13:50.477574 7f0149f3e700 10 mds.0.inotable: replay_reset [10000000000~10000000000]
2017-07-07 17:13:50.477585 7f0149f3e700 10 mds.0.inotable: skip_inos was [10000000000~10000000000]
2017-07-07 17:13:50.477594 7f0149f3e700 10 mds.0.inotable: skip_inos now [10000989680~ffff676980]

....
2017-07-07 17:20:54.994468 7f014c74a700 20 mds.0.scrubstack popping [inode 1 [...2,head] / auth v2 snaprealm=0x7f01619dfa80 f(v0 1=1+0) n() (iversion lock) | request=0 lock=0 dirfrag=1 caps=0 dirtyparent=0 scrubqueue=1 dirty=0 authpin=0 0x7f0161aee5d0 mode: 17407] off of ScrubStack
2017-07-07 17:20:54.994494 7f014ae46700 20 mds.0.log _submit_thread 4219550~512 : EUpdate repair_dirfrag [metablob 10000000000, 1 dirs]
2017-07-07 17:20:54.994499 7f014ae46700 10 mds.0.journaler(rw) append_entry len 512 to 4219550~532
2017-07-07 17:20:54.994503 7f014ae46700 20 mds.0.journaler(rw) flush not delaying flush
2017-07-07 17:20:54.994505 7f014ae46700 10 mds.0.journaler(rw) _do_flush flushing 4219550~532
2017-07-07 17:20:54.994533 7f014ae46700 10 mds.0.journaler(rw) _do_flush (prezeroing/prezero)/write/flush/safe pointers now at (29360128/29360128)/4220082/4220082/4214844
2017-07-07 17:20:54.994535 7f014ae46700 20 mds.0.journaler(rw) _issue_prezero target 29360128 <= prezeroing_pos 29360128
2017-07-07 17:20:54.995968 7f014c74a700 10 MDSIOContextBase::complete: 19MDSIOContextWrapper
2017-07-07 17:20:54.995974 7f014c74a700 10 mds.0.cache.ino(1) decoded 30 bytes of backtrace successfully
2017-07-07 17:20:54.995977 7f014c74a700 10 mds.0.cache.ino(1) scrub*: inotable ino = 0x1*
2017-07-07 17:20:54.995979 7f014c74a700 10 mds.0.cache.ino(1) scrub: inotable free says 0

If inode numbers were removed from inotable when replaying journal. how did "assert(inode_map.count(in->vino()) == 0)" happened, how did client session get prealloc inos [10000000001~3e8]?

Zheng Yan wrote:

If inode numbers were removed from inotable when replaying journal. how did "assert(inode_map.count(in->vino()) 0)" happened, how did client session get prealloc inos [10000000001~3e8]?

I did several times test case,but not reproduce ‘FAILED assert(inode_map.count(in->vino()) 0)’ and the ino of the file i touched is 1099521627776(10000989680).I think i might run 'cephfs-journal event splice --type=RESETJOURNAL summary' delete the journal entry or any operation else.If ‘FAILED assert(inode_map.count(in->vino()) == 0)’ happened again i will pay more attention on it.