Project

General

Profile

Bug #17939

non-local cephfs quota changes not visible until some IO is done

Added by Dan van der Ster 9 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Quotas
Target version:
-
Start date:
11/17/2016
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Release:
jewel
Component(FS):
ceph-fuse
Needs Doc:
No

Description

If we change the ceph.quota.max_bytes attribute on a cephfs mount, that quota is not applied until cephfs is remounted.

This affects Manila: where users can extend their shares, which changes the ceph.quota.max_bytes attr on a remote machine. But currently the users need to remount to see that new quota.

Reproducer:

# cat /etc/fstab  | grep ceph
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6    /cephfs    fuse.ceph    _netdev00
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6    /cephfs2    fuse.ceph    _netdev00

# mount /cephfs
# mount /cephfs2

# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        30G  104M   30G   1% /cephfs2

# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720" 

# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="32212254720" 

# cd /cephfs2
# setfattr -n ceph.quota.max_bytes -v 10737418240 .

# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        10G  104M  9.9G   2% /cephfs2

# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720" 

# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="10737418240" 

# echo 3 > /proc/sys/vm/drop_caches
# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        10G  104M  9.9G   2% /cephfs2

It would be better if that quota attrs would be periodically refreshed.


Related issues

Copied to fs - Backport #19762: jewel: non-local cephfs quota changes not visible until some IO is done Resolved
Copied to fs - Backport #19763: kraken: non-local cephfs quota changes not visible until some IO is done Resolved

History

#1 Updated by Ramana Raja 9 months ago

Hmmm. I remember testing manila's extend share API with cephfs's native driver, where the ceph_volume_client modifies the quota attributes. The client mount could see the new quota limits after a metadata IO that refreshed rstat. Please see,
https://github.com/ceph/ceph-qa-suite/commit/4e9635e8558aa69e66ec79b53469b03cd3eb12e6#diff-bfab91ebee6aab667c9783bbdde74edaR183

#2 Updated by Dan van der Ster 9 months ago

That commit seems to test client_quota_df. That works, I agree. This issue is about extending a share.

extend_share calls set_max_bytes:
https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/cephfs_native.py#L312

set_max_bytes only sets the xattr:
https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L1292

What is the rstat and where is it refreshed?

#3 Updated by Ramana Raja 9 months ago

  • Status changed from New to Need More Info

I don't think you need a remount (of '/cephfs' in your reproducer) for the new share quota limits to be reflected (for '/cephfs').

Can you please check in your reproducer that:
After setting new quota limits on '/cephfs2' (reduced from 30G to 10G) and e.g. creating a new file in '/cephfs2' or '/cephfs' (say, # sudo touch /cephfs2/foo),
the quota limits for both '/cephfs' and '/cephfs2' are correctly reflected (check with, # df -h | grep ceph)?

#4 Updated by Dan van der Ster 9 months ago

Thanks, indeed that works! I tried touch /cephfs/foo and touch /cephfs2/foo; in both cases the "wrong" quota was corrected. However, I also found that the quota is not updated if we touch a file in a subdir: touch /cephfs/subdir/foo does not fix the quotas. For now we can tell users about this workaround (touch a file in the root of /cephfs mount -- even just touch /cephfs works).

But I wonder if this is anyway a problem? Shouldn't the quota's be automatically updated without requiring users to write to the mount point?

#5 Updated by John Spray 9 months ago

  • Status changed from Need More Info to Verified

I can't immediately remember how synchronous reads on the quota xattrs are meant to be, but they should probably be doing a better job than what you're seeing. This will need investigation.

#6 Updated by Ramana Raja 9 months ago

  • Category changed from Manila to Quotas

#7 Updated by John Spray 9 months ago

  • Assignee changed from Ramana Raja to John Spray

#8 Updated by Dan van der Ster 6 months ago

I just realised that rstats have the same problem. Client A is adding data to a Manila share, and Client B doesn't see the parent's rstats change until we touch the parent on B.

#9 Updated by John Spray 5 months ago

Right, finally looking at this.

Turns out all our ceph.* vxattrs return whatever is in the local cache, with no regard for capabilities.

Quotas are protected on the server side by CInode::policylock, but I don't think there is a (at present) capability that the client can take to match that.

The reason we get the newly created quota after touching a file is that MDCache::predirty_journal_parents does a broadcast_quota_to_clients.

#10 Updated by John Spray 5 months ago

  • Subject changed from non-local cephfs quota changes require a remount to non-local cephfs quota changes not visible until some IO is done

#11 Updated by John Spray 5 months ago

  • Status changed from Verified to Need Review

#12 Updated by John Spray 4 months ago

  • Status changed from Need Review to Pending Backport
  • Backport set to jewel, kraken

#13 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #19762: jewel: non-local cephfs quota changes not visible until some IO is done added

#14 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #19763: kraken: non-local cephfs quota changes not visible until some IO is done added

#15 Updated by Nathan Cutler about 1 month ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF