Bug #17939
closed
non-local cephfs quota changes not visible until some IO is done
Added by Dan van der Ster over 7 years ago.
Updated about 5 years ago.
Description
If we change the ceph.quota.max_bytes attribute on a cephfs mount, that quota is not applied until cephfs is remounted.
This affects Manila: where users can extend their shares, which changes the ceph.quota.max_bytes attr on a remote machine. But currently the users need to remount to see that new quota.
Reproducer:
# cat /etc/fstab | grep ceph
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6 /cephfs fuse.ceph _netdev00
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6 /cephfs2 fuse.ceph _netdev00
# mount /cephfs
# mount /cephfs2
# df -h | grep ceph
ceph-fuse 30G 104M 30G 1% /cephfs
ceph-fuse 30G 104M 30G 1% /cephfs2
# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720"
# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="32212254720"
# cd /cephfs2
# setfattr -n ceph.quota.max_bytes -v 10737418240 .
# df -h | grep ceph
ceph-fuse 30G 104M 30G 1% /cephfs
ceph-fuse 10G 104M 9.9G 2% /cephfs2
# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720"
# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="10737418240"
# echo 3 > /proc/sys/vm/drop_caches
# df -h | grep ceph
ceph-fuse 30G 104M 30G 1% /cephfs
ceph-fuse 10G 104M 9.9G 2% /cephfs2
It would be better if that quota attrs would be periodically refreshed.
- Status changed from New to Need More Info
I don't think you need a remount (of '/cephfs' in your reproducer) for the new share quota limits to be reflected (for '/cephfs').
Can you please check in your reproducer that:
After setting new quota limits on '/cephfs2' (reduced from 30G to 10G) and e.g. creating a new file in '/cephfs2' or '/cephfs' (say, # sudo touch /cephfs2/foo),
the quota limits for both '/cephfs' and '/cephfs2' are correctly reflected (check with, # df -h | grep ceph)?
Thanks, indeed that works! I tried touch /cephfs/foo and touch /cephfs2/foo; in both cases the "wrong" quota was corrected. However, I also found that the quota is not updated if we touch a file in a subdir: touch /cephfs/subdir/foo does not fix the quotas. For now we can tell users about this workaround (touch a file in the root of /cephfs mount -- even just touch /cephfs works).
But I wonder if this is anyway a problem? Shouldn't the quota's be automatically updated without requiring users to write to the mount point?
- Status changed from Need More Info to 12
I can't immediately remember how synchronous reads on the quota xattrs are meant to be, but they should probably be doing a better job than what you're seeing. This will need investigation.
- Category changed from 87 to 97
- Assignee changed from Ramana Raja to John Spray
I just realised that rstats have the same problem. Client A is adding data to a Manila share, and Client B doesn't see the parent's rstats change until we touch the parent on B.
Right, finally looking at this.
Turns out all our ceph.* vxattrs return whatever is in the local cache, with no regard for capabilities.
Quotas are protected on the server side by CInode::policylock, but I don't think there is a (at present) capability that the client can take to match that.
The reason we get the newly created quota after touching a file is that MDCache::predirty_journal_parents does a broadcast_quota_to_clients.
- Subject changed from non-local cephfs quota changes require a remount to non-local cephfs quota changes not visible until some IO is done
- Status changed from 12 to Fix Under Review
- Status changed from Fix Under Review to Pending Backport
- Backport set to jewel, kraken
- Copied to Backport #19762: jewel: non-local cephfs quota changes not visible until some IO is done added
- Copied to Backport #19763: kraken: non-local cephfs quota changes not visible until some IO is done added
- Status changed from Pending Backport to Resolved
- Category deleted (
97)
- Labels (FS) quotas added
Also available in: Atom
PDF