https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2018-03-05T03:17:29Z
Ceph
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108445
2018-03-05T03:17:29Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>we haven't backported that fix</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108449
2018-03-05T04:01:06Z
lei gu
<ul></ul><p>I have backport it and make a full test.</p>
<p>The "Stale file handle" problem disappear, but it still have some problems( i based on v10.2.7):<br />1.I enable posix_acl, so I will get "Permission denied" error;<br />2.When I disable posix_acl, I will get "Not a directory" error.</p>
<p>Is there other relatied issues i ignored?</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108457
2018-03-05T08:59:34Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>looks like kernel tried to get parent for non-directory inode. which version of kernel do you use?</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108458
2018-03-05T09:05:18Z
Jos Collin
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108462
2018-03-05T10:25:23Z
lei gu
<ul></ul><p>My kernel release version is 3.10.0-327.el7.x86_64.</p>
<p>I have tried to find out what causes these error:<br />1."Permission denied" is related to posix_acl, which happens in function 'Inode::check_mode', it seems inode mode changed when mds migrating;<br />2."Not a directory" is happed in function 'Client::_lookup', because of 'dir->is_dir()' return false(looks like special directory ".").</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108464
2018-03-05T11:27:06Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>I checked code of 3.10.0-327.el7 kernel. It calls get_parent() for non-directory inode. Recent kernels only call get_parent() for directory inode.</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108465
2018-03-05T11:34:27Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>recent rhel kernel does not call get_parent() for non-directory inode. please upgrade your kernel</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108466
2018-03-05T11:44:12Z
lei gu
<ul></ul><p>OK, I will try and feedback later. Could you pleaese tell me which rhel kernel version changed the get_parent() call or do what's your kernel version?</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108664
2018-03-07T08:34:27Z
lei gu
<ul></ul><p>I have test the patch on kernel 4.15.0 using dd and vdbench. dd is OK, vdbench still have the "Stale file handle" problem.</p>
<p><strong>[vdbench config]</strong><br /><pre><code class="text syntaxhl"><span class="CodeRay">messagescan=no
hd=default,vdbench=/root/shawn/vdbench50406,user=root,shell=ssh
hd=hd1,system=192.169.30.210
fsd=default,depth=1,width=1,files=10000,size=512M,shared=no,openflag=o_direct
fsd=fsd1,anchor=/mnt/nfs
fwd=default,rdpct=60,xfersizes=2M,fileio=random, fileselect=sequential,threads=4
fwd=fwd1,fsd=fsd1,host=hd1
rd=rd1,fwd=fwd1,fwdrate=200,format=yes,warmup=5,elapsed=28800,interval=1
</span></code></pre></p>
<p><strong>[vdbech error info]</strong><br /><pre><code class="text syntaxhl"><span class="CodeRay">Write error using file /mnt/nfs/vdb.1_1.dir/vdb_f0060.file
Error: ESTALE: 'Stale NFS file handle'
lba: 340918272
xfersize: 131072
blocks_done: 2583
bytes_done: 338558976
open_for_read: false
fhandle: 21
</span></code></pre></p>
<p><strong>[ceph-fuse log]</strong><br /><pre><code class="text syntaxhl"><span class="CodeRay">2018-03-06 23:06:39.477988 7f07e75a2700 3 client.641150 lookup_ino enter(100000000c5) =
2018-03-06 23:06:39.478022 7f07e75a2700 10 client.641150 choose_target_mds resend_mds specified as mds.
0
2018-03-06 23:06:39.478027 7f07e75a2700 10 client.641150 send_request rebuilding request 487 for mds.0
2018-03-06 23:06:39.478033 7f07e75a2700 10 client.641150 send_request client_request(unknown.0:487 look
upino #100000000c5 2018-03-06 23:06:39.478018) v3 to mds.0
2018-03-06 23:06:39.478892 7f07f1cdc700 10 client.641150 insert_trace from 2018-03-06 23:06:39.478032 m
ds.0 is_target=1 is_dentry=0
2018-03-06 23:06:39.478897 7f07f1cdc700 10 client.641150 features 0x7ffffffefdfbfff
2018-03-06 23:06:39.478899 7f07f1cdc700 10 client.641150 update_snap_trace len 48
2018-03-06 23:06:39.478903 7f07f1cdc700 10 client.641150 update_snap_trace snaprealm(1 nref=55 c=0 seq=
1 parent=0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING
2018-03-06 23:06:39.478907 7f07f1cdc700 10 client.641150 hrm is_target=1 is_dentry=0
2018-03-06 23:06:39.478916 7f07f1cdc700 10 client.641150 update_inode_file_bits 100000000c5.head(faked_
ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100777 size=0/0 mtime=0.000000 caps=- objectset[100000000
c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300) - mtime 2018-03-06 22:59:43.709871
2018-03-06 23:06:39.478924 7f07f1cdc700 10 client.641150 size 0 -> 68
2018-03-06 23:06:39.478928 7f07f1cdc700 10 client.641150 add_update_cap issued - -> pAsLsXsFscr from md
s.0 on 100000000c5.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100777 size=68/0 mtime=2018
-03-06 22:59:43.709871 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_
tx 0] 0x555cd0846300)
2018-03-06 23:06:39.478967 7f07e75a2700 3 client.641150 lookup_ino exit(100000000c5) = 0
2018-03-06 23:06:39.478974 7f07e75a2700 3 client.641150 ll_lookup 0x555cd0846300 .
2018-03-06 23:06:39.478977 7f07e75a2700 10 client.641150 _getattr mask AsXs issued=1
2018-03-06 23:06:39.478983 7f07e75a2700 3 client.641150 may_lookup 0x555cd0846300 = 0
2018-03-06 23:06:39.478987 7f07e75a2700 10 client.641150 _lookup 100000000c5.head(faked_ino=0 ref=3 ll_
ref=1 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=pAsLsXsFscr(0=pAs
LsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300) . = -20
2018-03-06 23:06:39.478997 7f07e75a2700 3 client.641150 ll_lookup 0x555cd0846300 . -> -20 (0)
2018-03-06 23:06:39.479008 7f07f1cdc700 10 client.641150 put_inode on 100000000c5.head(faked_ino=0 ref=
3 ll_ref=1 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=pAsLsXsFscr(
0=pAsLsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300)
2018-03-06 23:06:39.479021 7f07f1cdc700 10 client.641150 put_inode on 100000000c5.head(faked_ino=0 ref=
2 ll_ref=1 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=pAsLsXsFscr(
0=pAsLsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300)
2018-03-06 23:06:39.479051 7f07e75a2700 3 client.641150 ll_forget 100000000c5 1
2018-03-06 23:06:39.479063 7f07e75a2700 10 client.641150 put_inode on 100000000c5.head(faked_ino=0 ref=
1 ll_ref=0 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=pAsLsXsFscr(
0=pAsLsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300)
2018-03-06 23:06:39.479073 7f07e75a2700 10 client.641150 remove_cap mds.0 on 100000000c5.head(faked_ino
=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=pAsLs
XsFscr(0=pAsLsXsFscr) objectset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300)
2018-03-06 23:06:39.479085 7f07e75a2700 10 client.641150 put_inode deleting 100000000c5.head(faked_ino=
0 ref=0 ll_ref=0 cap_refs={} open={} mode=100777 size=68/0 mtime=2018-03-06 22:59:43.709871 caps=- obje
ctset[100000000c5 ts 0/0 objects 0 dirty_or_tx 0] 0x555cd0846300)
</span></code></pre></p>
<p>I found "ll_lookup 0x555cd0846300 . -> -20 (0)". But I have no idea what happend.</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108690
2018-03-07T14:47:01Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Zheng Yan</i></li></ul>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108732
2018-03-08T03:23:47Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>could you try below patch<br /><pre>
diff --git a/src/client/Client.cc b/src/client/Client.cc
index b6bc15dbae..ae7877b2ea 100644
--- a/src/client/Client.cc
+++ b/src/client/Client.cc
@@ -6083,11 +6083,6 @@ int Client::_lookup(Inode *dir, const string& dname, int mask, InodeRef *target,
int r = 0;
Dentry *dn = NULL;
- if (!dir->is_dir()) {
- r = -ENOTDIR;
- goto done;
- }
-
if (dname == "..") {
if (dir->dentries.empty()) {
MetaRequest *req = new MetaRequest(CEPH_MDS_OP_LOOKUPPARENT);
@@ -6116,6 +6111,11 @@ int Client::_lookup(Inode *dir, const string& dname, int mask, InodeRef *target,
goto done;
}
+ if (!dir->is_dir()) {
+ r = -ENOTDIR;
+ goto done;
+ }
+
if (dname.length() > NAME_MAX) {
r = -ENAMETOOLONG;
goto done;
</pre></p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108736
2018-03-08T06:06:35Z
lei gu
<ul><li><strong>File</strong> <a href="/attachments/download/3306/ceph-client.admin.log.rar">ceph-client.admin.log.rar</a> added</li></ul><p>I have tested the patch, error log likes "ll_lookup 0x555cd0846300 . -> -20 (0)" disappeared, but there still have "Stale file handle" error.</p>
<p>I upload the log of ceph-fuse after mds migration.</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=108806
2018-03-09T01:10:56Z
lei gu
<ul></ul><p>May this is related to ceph fuse acl. When I disable "client_acl_type" and enable "fuse_default_permissions", everything works well.</p>
<p>My ceph.conf is as below:<br /><pre><code class="text syntaxhl"><span class="CodeRay">client_acl_type = posix_acl
fuse_default_permissions = false
</span></code></pre></p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=109095
2018-03-14T09:00:04Z
lei gu
<ul></ul><p>After debugging kernel(version 4.14), I found something in call relation "exportfs_decode_fh()->fuse_fh_to_dentry()->fuse_get_dentry()->fuse_lookup_name()" that a non-directory inode was passed.</p>
<p>I think this problem just likes issue 21995. Is there any suggestions?</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=109264
2018-03-19T01:51:52Z
lei gu
<ul></ul><p>I add some debug info in the call relation above. So I get the log both of kernel and ceph-fuse when error happend.</p>
<p>The size of the log is a little big, so I upload it in google drive which url is [[<a class="external" href="https://drive.google.com/open?id=1kdehTiqGaPnl1xH1BlxlLiRpFirz3uCA">https://drive.google.com/open?id=1kdehTiqGaPnl1xH1BlxlLiRpFirz3uCA</a>]]</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=109351
2018-03-20T12:31:36Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>but with above patch, "lookup ." on non-directory inode should works</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=109360
2018-03-20T12:49:07Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>try modifying Client::ll_lookup, ignore may_lookup check for lookup . and lookup ..</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=109494
2018-03-22T07:38:36Z
Jos Collin
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/21003">https://github.com/ceph/ceph/pull/21003</a></p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=110093
2018-04-03T18:41:12Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li><li><strong>Backport</strong> set to <i>luminous</i></li><li><strong>Component(FS)</strong> <i>Client</i> added</li><li><strong>Component(FS)</strong> deleted (<del><i>ceph-fuse</i></del>)</li></ul>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=110111
2018-04-04T06:05:07Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Pending Backport</i></li></ul><p>@Patrick, please confirm that this should be backported to luminous and which master PR.</p>
CephFS - Bug #23210: ceph-fuse: exported nfs get "stale file handle" when mds migrating
https://tracker.ceph.com/issues/23210?journal_id=110140
2018-04-04T17:51:35Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li><li><strong>Backport</strong> deleted (<del><i>luminous</i></del>)</li></ul><p>Sorry, this should not be backported. ceph-fuse "support" for NFS is only for Mimic.</p>