Bug #18013

ceph_test_libcephfs access failures when run without ceph-fuse mount in teuthology (no UserPerm?)

Added by Patrick Donnelly about 6 years ago. Updated about 6 years ago.

Target version:
% Done:


3 - minor
Affected Versions:
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):


When ceph-fuse is turned off for the libcephfs_interface_tests task, we see access control failures:

2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:Running main() from
2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[==========] Running 59 tests from 3 test cases.
2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[----------] Global test environment set-up.
2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[----------] 54 tests from LibCephFS
2016-11-22T19:56:38.947 INFO:tasks.workunit.client.0.mira056.stdout:[ RUN      ] LibCephFS.OpenEmptyComponent
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-1840-gf0b7d36/src/test/libcephfs/ Failure
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:      Expected: 0
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:To be equal to: ceph_mkdirs(cmount, c_dir, 0777)
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:      Which is: -13
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:[  FAILED  ] LibCephFS.OpenEmptyComponent (754 ms)
2016-11-22T19:56:39.701 INFO:tasks.workunit.client.0.mira056.stdout:[ RUN      ] LibCephFS.OpenReadWrite
2016-11-22T19:56:39.953 INFO:tasks.workunit.client.0.mira056.stdout:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-1840-gf0b7d36/src/test/libcephfs/ Failure
2016-11-22T19:56:39.954 INFO:tasks.workunit.client.0.mira056.stdout:Expected: (0) < (fd), actual: 0 vs -13
2016-11-22T19:56:39.954 INFO:tasks.workunit.client.0.mira056.stdout:[  FAILED  ] LibCephFS.OpenReadWrite (253 ms)


The first test failure gives this client log: /ceph/teuthology-archive/pdonnell-2016-11-22_19:29:40-fs-master-distro-basic-mira/570130/remote/mira056/log/ceph-client.admin.16057.log.gz

Key excerpt:

2016-11-22 19:56:39.700127 7f8d1c762e80 10 client.4137 Client::mkdirs /open_test_16057
2016-11-22 19:56:39.700151 7f8d1c762e80 10 client.4137 _getattr mask As issued=1
2016-11-22 19:56:39.700165 7f8d1c762e80  3 client.4137 may_lookup 0x7f8cec00b3c0 = 0
2016-11-22 19:56:39.700173 7f8d1c762e80 10 client.4137 _lookup concluded ENOENT locally for 1.head(faked_ino=0 ref=5 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 mtime=2016-11-22 19:54:33.020674 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7f8cec00b3c0) dn 'open_test_16057'
2016-11-22 19:56:39.700197 7f8d1c762e80 10 client.4137 _getattr mask As issued=1
2016-11-22 19:56:39.700201 7f8d1c762e80  3 client.4137 may_create 0x7f8cec00b3c0 = -13


#1 Updated by Patrick Donnelly about 6 years ago

BTW, the only material difference between this test run and a normal FS suite run is that ceph-fuse is turned off. Compare config to:

#2 Updated by Greg Farnum about 6 years ago

  • Subject changed from client: libcephfs access failures without ceph-fuse mount to ceph_test_libcephfs access failures when run without ceph-fuse mount in teuthology (no UserPerm?)
  • Category set to Testing
  • Status changed from New to 12
  • Component(FS) Client added
  • Component(FS) deleted (ceph-fuse)

Looks like these tests don't specify a UserPerm on the ceph function invocations, nor set up a "global" one. I imagine ceph-fuse is doing something to grant the "ubuntu" teuthology user permission to access these parts of the tree which otherwise get denied by default?

#3 Updated by John Spray about 6 years ago

This chmod may be the bit we're missing:

We could do a similar thing with ceph_ll_setattr and passing UserPerm constructed to be root?

#4 Updated by Jeff Layton about 6 years ago

  • Assignee set to Jeff Layton

#5 Updated by Jeff Layton about 6 years ago

I think the right solution is to add a new function for setting the default UserPerm for the cmount, and have those jobs set that when they run. I'll have a look at doing that soon... that I look, I like John's suggestion better -- do a ceph_ll_setattr as root, and change the permissions on the root to 01777. We might still want to add an interface to allow setting the default UserPerm on a mount, but I don't think that's really the right fix here.

#6 Updated by Jeff Layton about 6 years ago

Actually, now that I look further, it turns out that looks like it has just the function we need:

static int update_root_mode()                                                   
  struct ceph_mount_info *admin;                                                
  int r = ceph_create(&admin, NULL);                                            
  if (r < 0)                                                                    
    return r;                                                                   
  ceph_conf_read_file(admin, NULL);                                             
  ceph_conf_parse_env(admin, NULL);                                             
  ceph_conf_set(admin, "client_permissions", "false");                          
  r = ceph_mount(admin, "/");                                                   
  if (r < 0)                                                                    
    goto out;                                                                   
  r = ceph_chmod(admin, "/", 0777);                                             
  return r;                                                                     

Maybe the best thing would be to combine ceph_test_libcepfs and ceph_test_libcephfs_access into a single binary, and use the main() function defined in Then we can let it fix up the permissions at the start of the test run.

Hmmm...actually, a simple interim workaround might be to just reverse the order of the tests in qa/workunits/libcephfs/ ceph_test_libcephfs_access will fix permissions first thing, and that should persist when the other binary is run.

#7 Updated by Patrick Donnelly about 6 years ago

Sounds reasonable to me Jeff.

#8 Updated by Jeff Layton about 6 years ago

Potential fix up here. Still needs testing, but it seems to do the right thing on my box:

#9 Updated by Jeff Layton about 6 years ago

  • Status changed from 12 to Resolved

Fixed in commit 8a887edaffeed66f02f605ee78b9999ffd629a60.

Also available in: Atom PDF