Bug #48104
fs/ceph: ceph.dir.entries does not acquire necessary caps
0%
Description
The ceph-mgr asynchronously deletes a directory via libcephfs in the volumes plugin here:
2020-11-03T08:40:09.596+0000 7fbbd9bc4700 8 client.10692 rmdir(#0x10000000219/31bfb21d-b512-47a2-8dc0-c5de008f995e) = 0 2020-11-03T08:40:09.596+0000 7fbbd9bc4700 0 [volumes DEBUG volumes.fs.async_job] unregistering async job cephfs.b'31bfb21d-b512-47a2-8dc0-c5de008f995e' from thread <JobThread(puregejob.3, started daemon 140444788606720)>
From: /ceph/teuthology-archive/pdonnell-2020-11-03_04:01:15-fs:volumes-wip-pdonnell-testing-20201102.231317-distro-basic-smithi/5585695/remote/smithi140/log/ceph-mgr.y.log.gz
"0x10000000219" is /volumes/_deleting. The test then checks if the directory is empty:
2020-11-03T08:39:36.381 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting) 2020-11-03T08:39:36.424 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names 2020-11-03T08:39:36.425 INFO:teuthology.orchestra.run.smithi140.stdout:2 2020-11-03T08:39:37.232 INFO:teuthology.orchestra.run.smithi140:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-03T08:39:37.235 INFO:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-03T08:39:37.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-03T08:39:37.427 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting) 2020-11-03T08:39:37.464 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names 2020-11-03T08:39:37.465 INFO:teuthology.orchestra.run.smithi140.stdout:1 ... 2020-11-03T08:40:06.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-03T08:40:06.433 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting) 2020-11-03T08:40:06.465 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names 2020-11-03T08:40:06.466 INFO:teuthology.orchestra.run.smithi140.stdout:1
From: /ceph/teuthology-archive/pdonnell-2020-11-03_04:01:15-fs:volumes-wip-pdonnell-testing-20201102.231317-distro-basic-smithi/5585695/teuthology.log
It does not seem that the xattr lookup acquires the necessary caps to be cache coherent. I will convert the test to use readdir but the xattr code needs fixed.
Related issues
History
#1 Updated by Jeff Layton over 3 years ago
- Assignee set to Jeff Layton
#2 Updated by Jeff Layton over 3 years ago
This xattr is populated like this:
return ceph_fmt_xattr(val, size, "%lld", ci->i_files + ci->i_subdirs);
Basically, it just grabs a couple of int fields from the ceph_inode_info and shows that. We don't do any sort of cache coherency here, and instead rely on the MDS to push updates to us. If we don't have caps on the directory then I guess we don't get an update.
We can plumb in something that forces a request for Fs caps, which should make sure they're updated. Not sure what that will do to performance however.
#3 Updated by Xiubo Li over 3 years ago
Should we fix this in libcephfs too ?
#4 Updated by Jeff Layton over 3 years ago
Yes, most likely. That's on my to-do list, but I haven't gotten to it yet.
#5 Updated by Jeff Layton over 3 years ago
- Status changed from New to Fix Under Review
#6 Updated by Jeff Layton over 3 years ago
- Copied to Bug #48313: client: ceph.dir.entries does not acquire necessary caps added
#7 Updated by Jeff Layton over 2 years ago
- Status changed from Fix Under Review to In Progress
Not sure why I set this to "Fix Under Review" since we don't have a patch for this! What should we do here, do we really want a side-effect with reading the ceph.dir.entries xattr that causes it to grab caps for this?
What other vxattrs need similar treatment? We might as well fix them all...
#8 Updated by Jeff Layton over 1 year ago
- Assignee deleted (
Jeff Layton)