Bug #18013
closedceph_test_libcephfs access failures when run without ceph-fuse mount in teuthology (no UserPerm?)
0%
Description
When ceph-fuse is turned off for the libcephfs_interface_tests task, we see access control failures:
2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:Running main() from gmock_main.cc 2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[==========] Running 59 tests from 3 test cases. 2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[----------] Global test environment set-up. 2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[----------] 54 tests from LibCephFS 2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[ RUN ] LibCephFS.OpenEmptyComponent 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-1840-gf0b7d36/src/test/libcephfs/test.cc:47: Failure 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout: Expected: 0 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:To be equal to: ceph_mkdirs(cmount, c_dir, 0777) 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout: Which is: -13 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:[ FAILED ] LibCephFS.OpenEmptyComponent (754 ms) 2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:[ RUN ] LibCephFS.OpenReadWrite 2016-11-22T19:56:39.953 INFO:tasks.workunit.client.0.mira056.stdout:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-1840-gf0b7d36/src/test/libcephfs/test.cc:83: Failure 2016-11-22T19:56:39.954 INFO:tasks.workunit.client.0.mira056.stdout:Expected: (0) < (fd), actual: 0 vs -13 2016-11-22T19:56:39.954 INFO:tasks.workunit.client.0.mira056.stdout:[ FAILED ] LibCephFS.OpenReadWrite (253 ms)
from: http://pulpito.ceph.com/pdonnell-2016-11-22_19:29:40-fs-master-distro-basic-mira/570130/
The first test failure gives this client log: /ceph/teuthology-archive/pdonnell-2016-11-22_19:29:40-fs-master-distro-basic-mira/570130/remote/mira056/log/ceph-client.admin.16057.log.gz
Key excerpt:
2016-11-22 19:56:39.700127 7f8d1c762e80 10 client.4137 Client::mkdirs /open_test_16057 2016-11-22 19:56:39.700151 7f8d1c762e80 10 client.4137 _getattr mask As issued=1 2016-11-22 19:56:39.700165 7f8d1c762e80 3 client.4137 may_lookup 0x7f8cec00b3c0 = 0 2016-11-22 19:56:39.700173 7f8d1c762e80 10 client.4137 _lookup concluded ENOENT locally for 1.head(faked_ino=0 ref=5 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 mtime=2016-11-22 19:54:33.020674 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7f8cec00b3c0) dn 'open_test_16057' 2016-11-22 19:56:39.700197 7f8d1c762e80 10 client.4137 _getattr mask As issued=1 2016-11-22 19:56:39.700201 7f8d1c762e80 3 client.4137 may_create 0x7f8cec00b3c0 = -13
Updated by Patrick Donnelly over 7 years ago
BTW, the only material difference between this test run and a normal FS suite run is that ceph-fuse is turned off. Compare config to:
Updated by Greg Farnum over 7 years ago
- Subject changed from client: libcephfs access failures without ceph-fuse mount to ceph_test_libcephfs access failures when run without ceph-fuse mount in teuthology (no UserPerm?)
- Category set to Testing
- Status changed from New to 12
- Component(FS) Client added
- Component(FS) deleted (
ceph-fuse)
Looks like these tests don't specify a UserPerm on the ceph function invocations, nor set up a "global" one. I imagine ceph-fuse is doing something to grant the "ubuntu" teuthology user permission to access these parts of the tree which otherwise get denied by default?
Updated by John Spray over 7 years ago
This chmod may be the bit we're missing:
https://github.com/ceph/ceph-qa-suite/blob/master/tasks/cephfs/fuse_mount.py#L190
We could do a similar thing with ceph_ll_setattr and passing UserPerm constructed to be root?
Updated by Jeff Layton over 7 years ago
I think the right solution is to add a new function for setting the default UserPerm for the cmount, and have those jobs set that when they run. I'll have a look at doing that soon...
Hmm...now that I look, I like John's suggestion better -- do a ceph_ll_setattr as root, and change the permissions on the root to 01777. We might still want to add an interface to allow setting the default UserPerm on a mount, but I don't think that's really the right fix here.
Updated by Jeff Layton over 7 years ago
Actually, now that I look further, it turns out that access.cc looks like it has just the function we need:
static int update_root_mode() { struct ceph_mount_info *admin; int r = ceph_create(&admin, NULL); if (r < 0) return r; ceph_conf_read_file(admin, NULL); ceph_conf_parse_env(admin, NULL); ceph_conf_set(admin, "client_permissions", "false"); r = ceph_mount(admin, "/"); if (r < 0) goto out; r = ceph_chmod(admin, "/", 0777); out: ceph_shutdown(admin); return r; }
Maybe the best thing would be to combine ceph_test_libcepfs and ceph_test_libcephfs_access into a single binary, and use the main() function defined in access.cc. Then we can let it fix up the permissions at the start of the test run.
Hmmm...actually, a simple interim workaround might be to just reverse the order of the tests in qa/workunits/libcephfs/test.sh. ceph_test_libcephfs_access will fix permissions first thing, and that should persist when the other binary is run.
Updated by Jeff Layton over 7 years ago
Potential fix up here. Still needs testing, but it seems to do the right thing on my box:
Updated by Jeff Layton over 7 years ago
- Status changed from 12 to Resolved
Fixed in commit 8a887edaffeed66f02f605ee78b9999ffd629a60.