Project

General

Profile

Bug #48104

fs/ceph: ceph.dir.entries does not acquire necessary caps

Added by Patrick Donnelly over 3 years ago. Updated over 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

The ceph-mgr asynchronously deletes a directory via libcephfs in the volumes plugin here:

2020-11-03T08:40:09.596+0000 7fbbd9bc4700  8 client.10692 rmdir(#0x10000000219/31bfb21d-b512-47a2-8dc0-c5de008f995e) = 0
2020-11-03T08:40:09.596+0000 7fbbd9bc4700  0 [volumes DEBUG volumes.fs.async_job] unregistering async job cephfs.b'31bfb21d-b512-47a2-8dc0-c5de008f995e' from thread <JobThread(puregejob.3, started daemon 140444788606720)>

From: /ceph/teuthology-archive/pdonnell-2020-11-03_04:01:15-fs:volumes-wip-pdonnell-testing-20201102.231317-distro-basic-smithi/5585695/remote/smithi140/log/ceph-mgr.y.log.gz

"0x10000000219" is /volumes/_deleting. The test then checks if the directory is empty:

2020-11-03T08:39:36.381 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting)
2020-11-03T08:39:36.424 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names
2020-11-03T08:39:36.425 INFO:teuthology.orchestra.run.smithi140.stdout:2
2020-11-03T08:39:37.232 INFO:teuthology.orchestra.run.smithi140:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-11-03T08:39:37.235 INFO:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-11-03T08:39:37.426 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-11-03T08:39:37.427 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting)
2020-11-03T08:39:37.464 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names
2020-11-03T08:39:37.465 INFO:teuthology.orchestra.run.smithi140.stdout:1
...
2020-11-03T08:40:06.432 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-11-03T08:40:06.433 INFO:teuthology.orchestra.run.smithi140:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting)
2020-11-03T08:40:06.465 INFO:teuthology.orchestra.run.smithi140.stderr:getfattr: Removing leading '/' from absolute path names
2020-11-03T08:40:06.466 INFO:teuthology.orchestra.run.smithi140.stdout:1

From: /ceph/teuthology-archive/pdonnell-2020-11-03_04:01:15-fs:volumes-wip-pdonnell-testing-20201102.231317-distro-basic-smithi/5585695/teuthology.log

It does not seem that the xattr lookup acquires the necessary caps to be cache coherent. I will convert the test to use readdir but the xattr code needs fixed.


Related issues

Copied to CephFS - Bug #48313: client: ceph.dir.entries does not acquire necessary caps Resolved

History

#1 Updated by Jeff Layton over 3 years ago

  • Assignee set to Jeff Layton

#2 Updated by Jeff Layton over 3 years ago

This xattr is populated like this:

return ceph_fmt_xattr(val, size, "%lld", ci->i_files + ci->i_subdirs);

Basically, it just grabs a couple of int fields from the ceph_inode_info and shows that. We don't do any sort of cache coherency here, and instead rely on the MDS to push updates to us. If we don't have caps on the directory then I guess we don't get an update.

We can plumb in something that forces a request for Fs caps, which should make sure they're updated. Not sure what that will do to performance however.

#3 Updated by Xiubo Li over 3 years ago

Should we fix this in libcephfs too ?

#4 Updated by Jeff Layton over 3 years ago

Yes, most likely. That's on my to-do list, but I haven't gotten to it yet.

#5 Updated by Jeff Layton over 3 years ago

  • Status changed from New to Fix Under Review

#6 Updated by Jeff Layton over 3 years ago

  • Copied to Bug #48313: client: ceph.dir.entries does not acquire necessary caps added

#7 Updated by Jeff Layton over 2 years ago

  • Status changed from Fix Under Review to In Progress

Not sure why I set this to "Fix Under Review" since we don't have a patch for this! What should we do here, do we really want a side-effect with reading the ceph.dir.entries xattr that causes it to grab caps for this?

What other vxattrs need similar treatment? We might as well fix them all...

#8 Updated by Jeff Layton over 1 year ago

  • Assignee deleted (Jeff Layton)

Also available in: Atom PDF