Bug #34531
Updated by Patrick Donnelly over 5 years ago
cephfs does not appear in 'df' if the quota size is too small If the quota is 10000000 (bytes), the mount point appears in 'df': <pre> ceph-fuse 8.0M 0 8.0M 0% /srv/smb/winbak and 'mount': ceph-fuse on /srv/smb/winbak type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) </pre> If quota is 1000000, the mount point no longer appears in 'df', but does appear in 'mount'. On the mailing list, Zhi Zhang (David) says: " I think this should be caused by the calculation method of fsblkcnt_t in ceph-fuse. The total and used space will be right shifted by CEPH_BLOCK_SHIFT (22). So if the quota of a directory is less than 4MB, the total size after this calculation would be 0. Then 'df' commands won't report this mount point, but 'mount' command will still do. " Below is a debug session as suggested by John. I used quota 1000000 on the mount point. (Segfault occurred when I ctrl-C to kill process after initial mount.) <pre> 2018-08-17 14:34:54.952636 7f0e300b5140 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-fuse, pid 30502 ceph-fuse[30502]: starting ceph client 2018-08-17 14:34:54.958910 7f0e300b5140 -1 init, newargv = 0x556a3f060120 newargc=9 2018-08-17 14:34:54.961492 7f0e298a6700 10 client.0 ms_handle_connect on 128.104.164.197:6789/0 2018-08-17 14:34:54.965175 7f0e300b5140 10 client.18814183 Subscribing to map 'mdsmap' 2018-08-17 14:34:54.965198 7f0e300b5140 20 client.18814183 trim_cache size 0 max 16384 2018-08-17 14:34:54.966721 7f0e298a6700 1 client.18814183 handle_mds_map epoch 2336272 2018-08-17 14:34:54.966788 7f0e300b5140 20 client.18814183 populate_metadata read hostname 'tardis' 2018-08-17 14:34:54.966824 7f0e300b5140 10 client.18814183 did not get mds through better means, so chose random mds 0 2018-08-17 14:34:54.966826 7f0e300b5140 20 client.18814183 mds is 0 2018-08-17 14:34:54.966828 7f0e300b5140 10 client.18814183 _open_mds_session mds.0 2018-08-17 14:34:54.966858 7f0e300b5140 10 client.18814183 waiting for session to mds.0 to open 2018-08-17 14:34:54.969991 7f0e298a6700 10 client.18814183 ms_handle_connect on 10.128.198.59:6800/2643422990 2018-08-17 14:34:55.033974 7f0e298a6700 10 client.18814183 handle_client_session client_session(open) v1 from mds.0 2018-08-17 14:34:55.034030 7f0e298a6700 10 client.18814183 renew_caps mds.0 2018-08-17 14:34:55.034196 7f0e298a6700 10 client.18814183 connect_mds_targets for mds.0 2018-08-17 14:34:55.034269 7f0e300b5140 10 client.18814183 did not get mds through better means, so chose random mds 0 2018-08-17 14:34:55.034276 7f0e300b5140 20 client.18814183 mds is 0 2018-08-17 14:34:55.034280 7f0e300b5140 10 client.18814183 send_request rebuilding request 1 for mds.0 2018-08-17 14:34:55.034285 7f0e300b5140 20 client.18814183 encode_cap_releases enter (req: 0x556a3ee79200, mds: 0) 2018-08-17 14:34:55.034287 7f0e300b5140 20 client.18814183 send_request set sent_stamp to 2018-08-17 14:34:55.034287 2018-08-17 14:34:55.034291 7f0e300b5140 10 client.18814183 send_request client_request(unknown.0:1 getattr pAsLsXsFs #0x1/backups/winbak 2018-08-17 14:34:54.966812 caller_uid=0, caller_gid=0{}) v4 to mds.0 2018-08-17 14:34:55.034331 7f0e300b5140 20 client.18814183 awaiting reply|forward|kick on 0x7ffd3c6a1970 2018-08-17 14:34:55.035114 7f0e298a6700 10 client.18814183 handle_client_session client_session(renewcaps seq 1) v1 from mds.0 2018-08-17 14:34:55.035612 7f0e298a6700 20 client.18814183 handle_client_reply got a reply. Safe:1 tid 1 2018-08-17 14:34:55.035664 7f0e298a6700 10 client.18814183 insert_trace from 2018-08-17 14:34:55.034287 mds.0 is_target=1 is_dentry=0 2018-08-17 14:34:55.035707 7f0e298a6700 10 client.18814183 features 0x3ffddff8eea4fffb 2018-08-17 14:34:55.035744 7f0e298a6700 10 client.18814183 update_snap_trace len 48 2018-08-17 14:34:55.035783 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 0 -> 1 2018-08-17 14:34:55.035857 7f0e298a6700 10 client.18814183 update_snap_trace snaprealm(0x1 nref=1 c=0 seq=0 parent=0x0 my_snaps=[] cached_snapc=0=[]) seq 1 > 0 2018-08-17 14:34:55.035901 7f0e298a6700 10 client.18814183 invalidate_snaprealm_and_children snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[]) 2018-08-17 14:34:55.035962 7f0e298a6700 15 client.18814183 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[]) self|parent updated 2018-08-17 14:34:55.036004 7f0e298a6700 15 client.18814183 snapc 1=[] 2018-08-17 14:34:55.036041 7f0e298a6700 10 client.18814183 no new snap on snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[]) 2018-08-17 14:34:55.036079 7f0e298a6700 20 client.18814183 put_snap_realm 0x1 0x556a3ee5ea90 2 -> 1 2018-08-17 14:34:55.036117 7f0e298a6700 10 client.18814183 hrm is_target=1 is_dentry=0 2018-08-17 14:34:55.036182 7f0e298a6700 15 inode.get on 0x556a3f128000 0x100002291c4.head now 1 2018-08-17 14:34:55.036224 7f0e298a6700 12 client.18814183 add_update_inode adding 0x100002291c4.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40000 size=0/0 nlink=0 mtime=0.000000 caps=- 0x556a3f128000) caps pAsLsXsFs 2018-08-17 14:34:55.036285 7f0e298a6700 20 client.18814183 dir hash is 2 2018-08-17 14:34:55.036324 7f0e298a6700 10 client.18814183 update_inode_file_bits 0x100002291c4.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=0.000000 caps=- quota(max_bytes = 1000000 max_files = 0) 0x556a3f128000) - mtime 2018-08-15 14:09:02.547890 2018-08-17 14:34:55.036384 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 1 -> 2 2018-08-17 14:34:55.036423 7f0e298a6700 15 client.18814183 add_update_cap first one, opened snaprealm 0x556a3ee5ea90 2018-08-17 14:34:55.036462 7f0e298a6700 10 client.18814183 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x100002291c4.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-15 14:09:02.547890 caps=pAsLsXsFs(0=pAsLsXsFs) quota(max_bytes = 1000000 max_files = 0) 0x556a3f128000) 2018-08-17 14:34:55.036512 7f0e298a6700 15 inode.get on 0x556a3f128000 0x100002291c4.head now 2 2018-08-17 14:34:55.036550 7f0e298a6700 20 client.18814183 put_snap_realm 0x1 0x556a3ee5ea90 2 -> 1 2018-08-17 14:34:55.036587 7f0e298a6700 15 inode.get on 0x556a3f128000 0x100002291c4.head now 3 2018-08-17 14:34:55.036625 7f0e298a6700 20 client.18814183 handle_client_reply signalling caller 0x7ffd3c6a1970 2018-08-17 14:34:55.036668 7f0e298a6700 20 client.18814183 handle_client_reply awaiting kickback on tid 1 0x7f0e298a4c10 2018-08-17 14:34:55.036710 7f0e300b5140 20 client.18814183 sendrecv kickback on tid 1 0x7f0e298a4c10 2018-08-17 14:34:55.036717 7f0e300b5140 20 client.18814183 lat 0.002429 2018-08-17 14:34:55.036728 7f0e300b5140 10 client.18814183 did not get mds through better means, so chose random mds 0 2018-08-17 14:34:55.036731 7f0e300b5140 20 client.18814183 mds is 0 2018-08-17 14:34:55.036733 7f0e300b5140 10 client.18814183 send_request rebuilding request 2 for mds.0 2018-08-17 14:34:55.036735 7f0e300b5140 20 client.18814183 encode_cap_releases enter (req: 0x556a3ee79500, mds: 0) 2018-08-17 14:34:55.036737 7f0e300b5140 20 client.18814183 send_request set sent_stamp to 2018-08-17 14:34:55.036737 2018-08-17 14:34:55.036741 7f0e300b5140 10 client.18814183 send_request client_request(unknown.0:2 getattr pAsLsXsFs #0x1/backups 2018-08-17 14:34:55.036723 caller_uid=0, caller_gid=0{}) v4 to mds.0 2018-08-17 14:34:55.036766 7f0e300b5140 20 client.18814183 awaiting reply|forward|kick on 0x7ffd3c6a1970 2018-08-17 14:34:55.037241 7f0e298a6700 10 client.18814183 put_inode on 0x100002291c4.head(faked_ino=0 ref=3 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-15 14:09:02.547890 caps=pAsLsXsFs(0=pAsLsXsFs) quota(max_bytes = 1000000 max_files = 0) 0x556a3f128000) 2018-08-17 14:34:55.037302 7f0e298a6700 15 inode.put on 0x556a3f128000 0x100002291c4.head now 2 2018-08-17 14:34:55.037342 7f0e298a6700 10 client.18814183 put_inode on 0x100002291c4.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-15 14:09:02.547890 caps=pAsLsXsFs(0=pAsLsXsFs) quota(max_bytes = 1000000 max_files = 0) 0x556a3f128000) 2018-08-17 14:34:55.037387 7f0e298a6700 15 inode.put on 0x556a3f128000 0x100002291c4.head now 1 2018-08-17 14:34:55.037945 7f0e298a6700 20 client.18814183 handle_client_reply got a reply. Safe:1 tid 2 2018-08-17 14:34:55.037988 7f0e298a6700 10 client.18814183 insert_trace from 2018-08-17 14:34:55.036737 mds.0 is_target=1 is_dentry=0 2018-08-17 14:34:55.038032 7f0e298a6700 10 client.18814183 features 0x3ffddff8eea4fffb 2018-08-17 14:34:55.038068 7f0e298a6700 10 client.18814183 update_snap_trace len 48 2018-08-17 14:34:55.038106 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 1 -> 2 2018-08-17 14:34:55.038143 7f0e298a6700 10 client.18814183 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING 2018-08-17 14:34:55.038183 7f0e298a6700 10 client.18814183 hrm is_target=1 is_dentry=0 2018-08-17 14:34:55.038226 7f0e298a6700 15 inode.get on 0x556a3f128600 0x1000021a21a.head now 1 2018-08-17 14:34:55.038277 7f0e298a6700 12 client.18814183 add_update_inode adding 0x1000021a21a.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40000 size=0/0 nlink=0 mtime=0.000000 caps=- 0x556a3f128600) caps pAsLsXsFs 2018-08-17 14:34:55.038323 7f0e298a6700 20 client.18814183 dir hash is 2 2018-08-17 14:34:55.038360 7f0e298a6700 10 client.18814183 update_inode_file_bits 0x1000021a21a.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=0.000000 caps=- 0x556a3f128600) - mtime 2018-08-10 15:38:21.499518 2018-08-17 14:34:55.038407 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 2 -> 3 2018-08-17 14:34:55.038445 7f0e298a6700 15 client.18814183 add_update_cap first one, opened snaprealm 0x556a3ee5ea90 2018-08-17 14:34:55.038483 7f0e298a6700 10 client.18814183 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x1000021a21a.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-10 15:38:21.499518 caps=pAsLsXsFs(0=pAsLsXsFs) 0x556a3f128600) 2018-08-17 14:34:55.038530 7f0e298a6700 15 inode.get on 0x556a3f128600 0x1000021a21a.head now 2 2018-08-17 14:34:55.038567 7f0e298a6700 20 client.18814183 put_snap_realm 0x1 0x556a3ee5ea90 3 -> 2 2018-08-17 14:34:55.038604 7f0e298a6700 15 inode.get on 0x556a3f128600 0x1000021a21a.head now 3 2018-08-17 14:34:55.038642 7f0e298a6700 20 client.18814183 handle_client_reply signalling caller 0x7ffd3c6a1970 2018-08-17 14:34:55.038686 7f0e298a6700 20 client.18814183 handle_client_reply awaiting kickback on tid 2 0x7f0e298a4c10 2018-08-17 14:34:55.038728 7f0e300b5140 20 client.18814183 sendrecv kickback on tid 2 0x7f0e298a4c10 2018-08-17 14:34:55.038734 7f0e300b5140 20 client.18814183 lat 0.001997 2018-08-17 14:34:55.038743 7f0e300b5140 10 client.18814183 did not get mds through better means, so chose random mds 0 2018-08-17 14:34:55.038747 7f0e300b5140 20 client.18814183 mds is 0 2018-08-17 14:34:55.038749 7f0e300b5140 10 client.18814183 send_request rebuilding request 3 for mds.0 2018-08-17 14:34:55.038751 7f0e300b5140 20 client.18814183 encode_cap_releases enter (req: 0x556a3ee79800, mds: 0) 2018-08-17 14:34:55.038753 7f0e300b5140 20 client.18814183 send_request set sent_stamp to 2018-08-17 14:34:55.038752 2018-08-17 14:34:55.038756 7f0e300b5140 10 client.18814183 send_request client_request(unknown.0:3 getattr pAsLsXsFs #0x1 2018-08-17 14:34:55.038739 caller_uid=0, caller_gid=0{}) v4 to mds.0 2018-08-17 14:34:55.038782 7f0e300b5140 20 client.18814183 awaiting reply|forward|kick on 0x7ffd3c6a1970 2018-08-17 14:34:55.039064 7f0e298a6700 10 client.18814183 put_inode on 0x1000021a21a.head(faked_ino=0 ref=3 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-10 15:38:21.499518 caps=pAsLsXsFs(0=pAsLsXsFs) 0x556a3f128600) 2018-08-17 14:34:55.039114 7f0e298a6700 15 inode.put on 0x556a3f128600 0x1000021a21a.head now 2 2018-08-17 14:34:55.039153 7f0e298a6700 10 client.18814183 put_inode on 0x1000021a21a.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40555 size=0/0 nlink=1 mtime=2018-08-10 15:38:21.499518 caps=pAsLsXsFs(0=pAsLsXsFs) 0x556a3f128600) 2018-08-17 14:34:55.039198 7f0e298a6700 15 inode.put on 0x556a3f128600 0x1000021a21a.head now 1 2018-08-17 14:34:55.039890 7f0e298a6700 20 client.18814183 handle_client_reply got a reply. Safe:1 tid 3 2018-08-17 14:34:55.039954 7f0e298a6700 10 client.18814183 insert_trace from 2018-08-17 14:34:55.038752 mds.0 is_target=1 is_dentry=0 2018-08-17 14:34:55.040004 7f0e298a6700 10 client.18814183 features 0x3ffddff8eea4fffb 2018-08-17 14:34:55.040041 7f0e298a6700 10 client.18814183 update_snap_trace len 48 2018-08-17 14:34:55.040082 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 2 -> 3 2018-08-17 14:34:55.040120 7f0e298a6700 10 client.18814183 update_snap_trace snaprealm(0x1 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING 2018-08-17 14:34:55.040159 7f0e298a6700 10 client.18814183 hrm is_target=1 is_dentry=0 2018-08-17 14:34:55.040211 7f0e298a6700 15 inode.get on 0x556a3f128c00 0x1.head now 1 2018-08-17 14:34:55.040251 7f0e298a6700 12 client.18814183 add_update_inode adding 0x1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40000 size=0/0 nlink=0 mtime=0.000000 caps=- 0x556a3f128c00) caps pAsLsXsFs 2018-08-17 14:34:55.040294 7f0e298a6700 20 client.18814183 dir hash is 2 2018-08-17 14:34:55.040331 7f0e298a6700 10 client.18814183 update_inode_file_bits 0x1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=0.000000 caps=- has_dir_layout 0x556a3f128c00) - mtime 2018-05-16 08:33:31.388505 2018-08-17 14:34:55.040376 7f0e298a6700 20 client.18814183 get_snap_realm 0x1 0x556a3ee5ea90 3 -> 4 2018-08-17 14:34:55.040414 7f0e298a6700 15 client.18814183 add_update_cap first one, opened snaprealm 0x556a3ee5ea90 2018-08-17 14:34:55.040452 7f0e298a6700 10 client.18814183 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2018-05-16 08:33:31.388505 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x556a3f128c00) 2018-08-17 14:34:55.040499 7f0e298a6700 15 inode.get on 0x556a3f128c00 0x1.head now 2 2018-08-17 14:34:55.040536 7f0e298a6700 20 client.18814183 put_snap_realm 0x1 0x556a3ee5ea90 4 -> 3 2018-08-17 14:34:55.040573 7f0e298a6700 15 inode.get on 0x556a3f128c00 0x1.head now 3 2018-08-17 14:34:55.040610 7f0e298a6700 20 client.18814183 handle_client_reply signalling caller 0x7ffd3c6a1970 ceph-fuse[30502]: starting fuse 2018-08-17 14:34:55.040653 7f0e298a6700 20 client.18814183 handle_client_reply awaiting kickback on tid 3 0x7f0e298a4c10 2018-08-17 14:34:55.040664 7f0e300b5140 20 client.18814183 sendrecv kickback on tid 3 0x7f0e298a4c10 2018-08-17 14:34:55.040667 7f0e300b5140 20 client.18814183 lat 0.001914 2018-08-17 14:34:55.040670 7f0e300b5140 15 inode.get on 0x556a3f128000 0x100002291c4.head now 2 2018-08-17 14:34:55.040672 7f0e300b5140 20 client.18814183 _ll_get 0x556a3f128000 0x100002291c4 -> 1 2018-08-17 14:34:55.041035 7f0e300b5140 10 client.18814183 ll_register_callbacks cb 0x556a3ee82c80 invalidate_ino_cb 1 invalidate_dentry_cb 1 switch_interrupt_cb 1 remount_cb 1 2018-08-17 14:34:55.048403 7f0e298a6700 10 client.18814183 put_inode on 0x1.head(faked_ino=0 ref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2018-05-16 08:33:31.388505 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x556a3f128c00) 2018-08-17 14:34:55.048421 7f0e298a6700 15 inode.put on 0x556a3f128c00 0x1.head now 2 2018-08-17 14:34:55.048424 7f0e298a6700 10 client.18814183 put_inode on 0x1.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2018-05-16 08:33:31.388505 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x556a3f128c00) 2018-08-17 14:34:55.048429 7f0e298a6700 15 inode.put on 0x556a3f128c00 0x1.head now 1 2018-08-17 14:34:55.051029 7f0e2589e700 1 client.18814183 using remount_cb 2018-08-17 14:34:55.055053 7f0e2509d700 3 client.18814183 ll_getattr 0x100002291c4.head 2018-08-17 14:34:55.055070 7f0e2509d700 10 client.18814183 _getattr mask pAsLsXsFs issued=1 2018-08-17 14:34:55.055074 7f0e2509d700 10 client.18814183 fill_stat on 0x100002291c4 snap/devhead mode 040555 mtime 2018-08-15 14:09:02.547890 ctime 2018-08-17 14:28:09.654639 2018-08-17 14:34:55.055089 7f0e2509d700 3 client.18814183 ll_getattr 0x100002291c4.head = 0 2018-08-17 14:34:55.055100 7f0e2509d700 3 client.18814183 ll_forget 0x100002291c4 1 2018-08-17 14:34:55.055102 7f0e2509d700 20 client.18814183 _ll_put 0x556a3f128000 0x100002291c4 1 -> 1 2018-08-17 14:34:55.965416 7f0e2a8a8700 10 client.18814183 renew_caps() 2018-08-17 14:34:55.965432 7f0e2a8a8700 15 client.18814183 renew_caps requesting from mds.0 2018-08-17 14:34:55.965436 7f0e2a8a8700 10 client.18814183 renew_caps mds.0 2018-08-17 14:34:55.965504 7f0e2a8a8700 20 client.18814183 trim_cache size 0 max 16384 2018-08-17 14:34:55.967114 7f0e298a6700 10 client.18814183 handle_client_session client_session(renewcaps seq 2) v1 from mds.0 ceph-fuse[30502]: fuse finished with error 0 and tester_r 0 *** Caught signal (Segmentation fault) ** </pre>