Ceph

/a/sage-2015-08-30_05:52:47-rados-master---basic-multi/1039290

i spot this failure also /a/kchai-2015-08-30_20:25:51-rados-master---basic-multi/1039521/. it failed with the first test,

rados -p "${POOL}" set-alloc-hint "${OBJ}" "${SMALL_HINT}" "${SMALL_HINT}" 
expect_alloc_hint_eq "${SMALL_HINT}" 

but the tests passed in local env.

CEPH_DEV_DIR=/media/xfs/dev MON=1 OSD=3 ./vstart.sh -n -l -X
../qa/workunits/rados/test_alloc_hint.sh

sage suspects

it could be caused by "permission denied": "ubuntu" failed to read osd's data dir, which is owned by "ceph:ceph".

http://pulpito.ceph.com/kchai-2015-09-07_23:23:54-rados-wip-kefu-testing---basic-multi/1046356/ failed.

2015-09-07T23:29:47.784 INFO:tasks.workunit.client.0.burnupi26.stderr:++ sudo find -type f
2015-09-07T23:29:47.785 INFO:tasks.workunit.client.0.burnupi26.stderr:++ grep head/foo_
2015-09-07T23:29:47.792 INFO:tasks.workunit.client.0.burnupi26.stderr:+ local fns=
2015-09-07T23:29:47.792 INFO:tasks.workunit.client.0.burnupi26.stderr:+ local count=1
2015-09-07T23:29:47.792 INFO:tasks.workunit.client.0.burnupi26.stderr:+ '[' 1 -ne 1 ']'
2015-09-07T23:29:47.792 INFO:tasks.workunit.client.0.burnupi26.stderr:+ local extsize
2015-09-07T23:29:47.793 INFO:tasks.workunit.client.0.burnupi26.stderr:++ sudo xfs_io -c extsize ''
2015-09-07T23:29:47.856 INFO:tasks.workunit.client.0.burnupi26.stderr:No such file or directory
2015-09-07T23:29:47.857 INFO:tasks.workunit.client.0.burnupi26.stderr:+ extsize=

sha1 of the wip-kefu-testing branch: 6363af54296a5eddfeaa62af852e99e35816f56a

$ git merge-base --is-ancestor 64962aafed362a2a798eefe54158f65af767f0bc 6363af54296a5eddfeaa62af852e99e35816f56a && echo 1
1
$ git show --oneline 64962aafed362a2a798eefe54158f65af767f0bc
64962aa qa/workunits/rados/test_alloc_hint.sh: sudo to list files
diff --git a/qa/workunits/rados/test_alloc_hint.sh b/qa/workunits/rados/test_alloc_hint.sh
index 86d3986..c43fc3c 100755
--- a/qa/workunits/rados/test_alloc_hint.sh
+++ b/qa/workunits/rados/test_alloc_hint.sh
@@ -61,7 +61,7 @@ function expect_alloc_hint_eq() {

         # e.g., .../25.6_head/foo__head_7FC1F406__19
         #       .../26.bs1_head/bar__head_EFE6384B__1a_ffffffffffffffff_1
-        local fns=(${OSD_DATA[i]}/current/${PGID}*_head/${OBJ}_*)
+        local fns=$(sudo find ${OSD_DATA[i]}/current/${PGID}*_head -type f | grep head/${OBJ}_)
         local count="${#fns[@]}" 
         if [ "${count}" -ne 1 ]; then
             echo "bad fns count: ${count}" >&2

in ceph.git/wip-12861. in the test script, i added a "sleep 2h" after flushing the object, will try to login to the test machine to see what happened.

# find /var/lib/ceph/osd/ceph-0/current/1.6*_head -type f | grep head/foo_
/var/lib/ceph/osd/ceph-0/current/1.6_head/foo__head_7FC1F406__1

$ sudo find /var/lib/ceph/osd/ceph-0/current/1.6*_head -type f | grep head/foo_
find: `/var/lib/ceph/osd/ceph-0/current/1.6*_head': No such file or directory
$ fns=$(sudo find /var/lib/ceph/osd/ceph-0/current/1.6*_head -type f | grep head/foo_)
find: `/var/lib/ceph/osd/ceph-0/current/1.6*_head': No such file or directory
$ echo $fns

$ echo "${#fns[@]}" 
1

probably we should go this (ugly) way:

$ fns=$(sudo sh -c 'ls /var/lib/ceph/osd/ceph-0/current/1.6*_head/foo_*')
$ echo $fns
/var/lib/ceph/osd/ceph-0/current/1.6_head/foo__head_7FC1F406__1