Ceph » CephFS

mpi-fsx also gets EPERM.

kclient and MDS never return -EACCES. was ior executed with root privilege?

saw similar error on ceph-fuse:

2013-07-17T12:35:23.001 INFO:teuthology.orchestra.run.out:access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
2013-07-17T12:35:23.001 INFO:teuthology.orchestra.run.out:------    ---------  ---------- ---------  --------   --------   --------   --------   ----
2013-07-17T12:35:23.340 INFO:teuthology.orchestra.run.out:ior ERROR: open64() failed, errno 13, Permission denied (aiori-POSIX.c:158)
2013-07-17T12:35:23.340 INFO:teuthology.orchestra.run.err:application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:Fatal error in PMPI_Reduce: Other MPI error, error stack:
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:PMPI_Reduce(1270)...............: MPI_Reduce(sbuf=0x7fff84fe1ffc, rbuf=0x7fff84fe2008, count=1, MPI_INT, MPI_SUM, root=0, comm=0x84000000) failed
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:MPIR_Reduce_impl(1087)..........:
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:MPIR_Reduce_intra(895)..........:
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:MPIR_Reduce_binomial(144).......:
2013-07-17T12:35:23.843 INFO:teuthology.orchestra.run.err:MPIDI_CH3U_Recvq_FDU_or_AEP(380): Communication error with rank 2

ubuntu@teuthology:/a/teuthology-2013-07-17_01:01:03-fs-next-testing-basic/70462$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 77c8bf2f972a9d6ff446c49a41678bf931bbee44
machine_type: plana
nuke-on-error: true
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 884fa2fcb6d707b23317bab1da909586ddc27608
  ceph-deploy:
    conf:
      client:
        debug monc: 20
        debug ms: 1
        debug objecter: 20
        debug rados: 20
        log file: /var/log/ceph/ceph-..log
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 884fa2fcb6d707b23317bab1da909586ddc27608
  s3tests:
    branch: next
  workunit:
    sha1: 884fa2fcb6d707b23317bab1da909586ddc27608
roles:
- - mon.a
  - mon.b
  - mon.c
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - client.2
- - client.1
- - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph: null
- ceph-fuse: null
- pexec:
    clients:
    - cd $TESTDIR
    - wget http://ceph.com/qa/ior.tbz2
    - tar xvfj ior.tbz2
    - cd ior
    - ./configure
    - make
    - make install DESTDIR=$TESTDIR/binary/
    - cd $TESTDIR/
    - rm ior.tbz2
    - rm -r ior
    - ln -s $TESTDIR/mnt.* $TESTDIR/gmnt
- ssh_keys: null
- mpi:
    exec: $TESTDIR/binary/usr/local/bin/ior -e -w -r -W -b 10m -a POSIX -o $TESTDIR/gmnt/ior.testfile
- pexec:
    all:
    - rm -f $TESTDIR/gmnt/ior.testfile
    - rm -f $TESTDIR/gmnt
    - rm -rf $TESTDIR/binary

Zheng Yan wrote:

kclient and MDS never return -EACCES. was ior executed with root privilege?

not root...

can't reproduce locally. Is it possible your clients have different 'username -> uid' mapping?

ah, a few do! the ones this test failed on don't, but maybe they were reimaged since then.