Bug #17939: non-local cephfs quota changes not visible until some IO is done - CephFS - Ceph

Actions

Copy link

Bug #17939

closed

non-local cephfs quota changes not visible until some IO is done

Added by Dan van der Ster over 7 years ago. Updated about 5 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

John Spray

Category:

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

jewel, kraken

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v10.2.3

ceph-qa-suite:

Component(FS):

ceph-fuse

Labels (FS):

quotas

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

If we change the ceph.quota.max_bytes attribute on a cephfs mount, that quota is not applied until cephfs is remounted.

This affects Manila: where users can extend their shares, which changes the ceph.quota.max_bytes attr on a remote machine. But currently the users need to remount to see that new quota.

Reproducer:

# cat /etc/fstab  | grep ceph
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6    /cephfs    fuse.ceph    _netdev00
conf:/etc/ceph/ceph.conf,id:dvanders,client-quota,client-mountpoint:/volumes/_nogroup/606824b2-1478-4aaa-9c08-4bcdf47202e6    /cephfs2    fuse.ceph    _netdev00

# mount /cephfs
# mount /cephfs2

# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        30G  104M   30G   1% /cephfs2

# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720" 

# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="32212254720" 

# cd /cephfs2
# setfattr -n ceph.quota.max_bytes -v 10737418240 .

# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        10G  104M  9.9G   2% /cephfs2

# getfattr -n ceph.quota.max_bytes /cephfs
getfattr: Removing leading '/' from absolute path names
# file: cephfs
ceph.quota.max_bytes="32212254720" 

# getfattr -n ceph.quota.max_bytes /cephfs2
getfattr: Removing leading '/' from absolute path names
# file: cephfs2
ceph.quota.max_bytes="10737418240" 

# echo 3 > /proc/sys/vm/drop_caches
# df -h | grep ceph
ceph-fuse        30G  104M   30G   1% /cephfs
ceph-fuse        10G  104M  9.9G   2% /cephfs2

It would be better if that quota attrs would be periodically refreshed.

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Ramana Raja over 7 years ago

Hmmm. I remember testing manila's extend share API with cephfs's native driver, where the ceph_volume_client modifies the quota attributes. The client mount could see the new quota limits after a metadata IO that refreshed rstat. Please see,
https://github.com/ceph/ceph-qa-suite/commit/4e9635e8558aa69e66ec79b53469b03cd3eb12e6#diff-bfab91ebee6aab667c9783bbdde74edaR183

Actions

Copy link

Updated by Dan van der Ster over 7 years ago

That commit seems to test client_quota_df. That works, I agree. This issue is about extending a share.

extend_share calls set_max_bytes:
https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/cephfs_native.py#L312

set_max_bytes only sets the xattr:
https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L1292

What is the rstat and where is it refreshed?

Actions

Copy link

Updated by Ramana Raja over 7 years ago

Status changed from New to Need More Info

I don't think you need a remount (of '/cephfs' in your reproducer) for the new share quota limits to be reflected (for '/cephfs').

Can you please check in your reproducer that:
After setting new quota limits on '/cephfs2' (reduced from 30G to 10G) and e.g. creating a new file in '/cephfs2' or '/cephfs' (say, # sudo touch /cephfs2/foo),
the quota limits for both '/cephfs' and '/cephfs2' are correctly reflected (check with, # df -h | grep ceph)?

Actions

Copy link

Updated by Dan van der Ster over 7 years ago

Thanks, indeed that works! I tried touch /cephfs/foo and touch /cephfs2/foo; in both cases the "wrong" quota was corrected. However, I also found that the quota is not updated if we touch a file in a subdir: touch /cephfs/subdir/foo does not fix the quotas. For now we can tell users about this workaround (touch a file in the root of /cephfs mount -- even just touch /cephfs works).

But I wonder if this is anyway a problem? Shouldn't the quota's be automatically updated without requiring users to write to the mount point?

Actions

Copy link

Updated by John Spray over 7 years ago

Status changed from Need More Info to 12

I can't immediately remember how synchronous reads on the quota xattrs are meant to be, but they should probably be doing a better job than what you're seeing. This will need investigation.

Actions

Copy link

Updated by Ramana Raja over 7 years ago

Category changed from 87 to 97

Actions

Copy link

Updated by John Spray over 7 years ago

Assignee changed from Ramana Raja to John Spray

Actions

Copy link

Updated by Dan van der Ster about 7 years ago

I just realised that rstats have the same problem. Client A is adding data to a Manila share, and Client B doesn't see the parent's rstats change until we touch the parent on B.

Actions

Copy link

Updated by John Spray about 7 years ago

Right, finally looking at this.

Turns out all our ceph.* vxattrs return whatever is in the local cache, with no regard for capabilities.

Quotas are protected on the server side by CInode::policylock, but I don't think there is a (at present) capability that the client can take to match that.

The reason we get the newly created quota after touching a file is that MDCache::predirty_journal_parents does a broadcast_quota_to_clients.

Actions

Copy link

#10