Ceph

It's most probably because of something that changed in https://github.com/ceph/ceph/pull/5699. How did you run this test ?

./virtualenv/bin/teuthology-suite --priority 50 --suite smoke/1node --suite-branch wip-11881-multipath --distro ubuntu --email loic@dachary.org --ceph infernalis --machine-type plana,burnupi,mira

diff --git a/tasks/ceph_deploy.py b/tasks/ceph_deploy.py
index 26d320a..6b62ab4 100644
--- a/tasks/ceph_deploy.py
+++ b/tasks/ceph_deploy.py
@@ -528,2 +528,3 @@ def cli_test(ctx, config):

+    time.sleep(1000000000)
     for i in range(3):

root@burnupi26:~# ceph-disk activate /dev/sdb1
got monmap epoch 1
2015-08-31 12:34:29.682753 7f99d8f70980 -1 filestore(/var/lib/ceph/tmp/mnt.xhhC1E) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.xhhC1E/journal: (13) Permission denied
2015-08-31 12:34:29.682795 7f99d8f70980 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
2015-08-31 12:34:29.682894 7f99d8f70980 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.xhhC1E: (13) Permission denied
ERROR:ceph-disk:Failed to activate
Traceback (most recent call last):
  File "/usr/sbin/ceph-disk", line 3326, in <module>
    main(sys.argv[1:])
  File "/usr/sbin/ceph-disk", line 3285, in main
    main_catch(args.func, args)
  File "/usr/sbin/ceph-disk", line 3304, in main_catch
    func(args)
  File "/usr/sbin/ceph-disk", line 2386, in main_activate
    dmcrypt_key_dir=args.dmcrypt_key_dir,
  File "/usr/sbin/ceph-disk", line 2159, in mount_activate
    (osd_id, cluster) = activate(path, activate_key_template, init)
  File "/usr/sbin/ceph-disk", line 2322, in activate
    keyring=keyring,
  File "/usr/sbin/ceph-disk", line 1925, in mkfs
    '--setgroup', get_ceph_user(),
  File "/usr/sbin/ceph-disk", line 333, in command_check_call
    return subprocess.check_call(arguments)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '0', '--monmap', '/var/lib/ceph/tmp/mnt.xhhC1E/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.xhhC1E', '--osd-journal', '/var/lib/ceph/tmp/mnt.xhhC1E/journal', '--osd-uuid', '616bb8bf-b2b7-4b8b-b127-67466f7e32e6', '--keyring', '/var/lib/ceph/tmp/mnt.xhhC1E/keyring', '--setuser', 'ceph', '--setgroup', 'ceph']' returned non-zero exit status 1

@Vasu Kulkarni it looks like it's related to '--setuser', 'ceph', '--setgroup', 'ceph' which comes from https://github.com/ceph/ceph/pull/4456

Sage is doing something right now at https://github.com/liewegas/ceph/commits/wip-user (https://github.com/liewegas/ceph/commit/f31baaa315327a437a861243dfb1c31de9550ef7)

I think what happens is the following:

• ceph-disk prepare triggers a udev event
• the udev rule calls ceph-disk activate as root and mounts the file system at a temporary location as root
• ceph-disk is called as ceph and fails because the mount point is owned by root

It looks like https://github.com/liewegas/ceph/commit/f31baaa315327a437a861243dfb1c31de9550ef7 can fix this.

It's simpler actually:

<sage> it's because the block device isn't owned by ceph.ceph.. that's the first udev rule fix in wip-user

See https://github.com/ceph/ceph/pull/5727 for a larger fix including the permission fix.