Ceph » CephFS

A task that implements this could be useful for testing calamari as well (I manually did some of the things needed here and generated some Cluster Updates are Stale messages on the gui).

A few questions about this:

1. Does the following seem like a reasonable way to go?
    a. a new task, multiclient_io, will create a random file on each client.
    b. rados -p rbd put xxx yyy will be run on each client.
    c. Each client will then do rados gets of every other client, and files can be
       compared (sha1 values to make this quicker).

2. Will the write(2) EOF and truncation testing involve using a work unit written
   in C that will do the write?  Or is it possible to do this using python?

3. This report is 3 years old.  In the meantime, has any of this been implemented
   somewhere else?  I have not found it, although it seems like some of this may
   end up getting tested by some of the other tasks.

A first pass of this is in origin/wip-multiclientio-wusui

Answering my own question: Item 2 above. It looks like this can all be done from python.

Note to self:
Try: rbd import to create an image name, rbd resize the image, make sure reads return EOF at right time for the new size.

# make sure we get the same MPI version on all hosts
os_type: ubuntu
os_version: "14.04" 

tasks:
- pexec:
    clients:
      - cd $TESTDIR
      - wget http://ceph.com/qa/fsx-mpi.c
      - mpicc fsx-mpi.c -o fsx-mpi
      - rm fsx-mpi.c
      - ln -s $TESTDIR/mnt.* $TESTDIR/gmnt
- ssh_keys:
- mpi:
    exec: $TESTDIR/fsx-mpi 1MB -N 50000 -p 10000 -l 1048576
    workdir: $TESTDIR/gmnt
- pexec:
    all:
      - rm $TESTDIR/gmnt
      - rm $TESTDIR/fsx-mpi

I'm not sure if that symlink created is correct. It produces a symlink to a non-existent mnt.* file. I think that we need to have one mnt.* file or we need to have the gmnt directory already made by then.

The problem i believe is that we need to install ceph and make sure that we have some mount points before we run the multiclient io test.

Currently I am testing with the following yaml file.

interactive-on-error: true
roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
- [mon.b, mds.a, osd.3, osd.4, osd.5, client.1]
tasks:
- chef:                                                              (New line)
- install:                                                           (New line)
- pexec:
    clients:
      - cd $TESTDIR
      - wget http://ceph.com/qa/fsx-mpi.c
      - mpicc fsx-mpi.c -o fsx-mpi
      - rm fsx-mpi.c
      - mkdir $TESTDIR/mnt.0                                         (New line)
      - ln -s $TESTDIR/mnt.* $TESTDIR/gmnt
- ssh_keys:
- mpi:
    exec: $TESTDIR/fsx-mpi 1MB -N 50000 -p 10000 -l 1048576
    workdir: $TESTDIR/gmnt
- pexec:
    all:
      - rm $TESTDIR/gmnt
      - rm $TESTDIR/fsx-mpi

First change: added
- chef:
- install:

just to make sure as much is installed as this should need. At this point, the
test failed with:

2014-12-09T15:48:20.103 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on vpm114 with status 1: 'sudo chmod 777 /var/log/ceph'
2014-12-09T15:48:20.107 INFO:teuthology.run:Summary data:
{duration: 1789.037582874298, failure_reason: 'Command failed on vpm114 with status
    1: ''sudo chmod 777 /var/log/ceph''', owner: wusui@aardvark, status: fail, success: false}

So I added the mkdir line above as an experiment. At this point, I got:

2014-12-09T18:01:19.912 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on vpm055 with status 91: 'mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt /home/ubuntu/cephtest/fsx-mpi 1MB -N 50000 -p 10000 -l 1048576'
2014-12-09T18:01:19.918 INFO:teuthology.run:Summary data:
{duration: 3158.258481025696, failure_reason: 'Command failed on vpm055 with status
    91: ''mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt
    /home/ubuntu/cephtest/fsx-mpi 1MB -N 50000 -p 10000 -l 1048576''', flavor: basic,

mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt /home/ubuntu/cephtest/fsx-mpi 1B -N 50000 -p 10000 -l 1048576

and I got:

1B: No such file or directory
Fatal error in PMPI_Barrier: Other MPI error, error stack:
PMPI_Barrier(425).........: MPI_Barrier(MPI_COMM_WORLD) failed
MPIR_Barrier_impl(331)....: Failure during collective
MPIR_Barrier_impl(313)....: 
MPIR_Barrier_intra(83)....: 
dequeue_and_set_error(596): Communication error with rank 1

So It looks like the syntax of the command is wrong (1B looks like it should be a number not a file to me).

TODO: look at http://ceph.com/qa/fsx-mpi.c to try to figure this out.

Here's what looks to be the last issue:

mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt /home/ubuntu/cephtest/fsx-mpi 1B -N 50000 -p 10000 -l 1048576

works when written as:

mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt sudo /home/ubuntu/cephtest/fsx-mpi -o 1B -N 50000 -p 10000 -l 1048576 /home/ubuntu/cephtest/gmnt/test

It appears that there is a missing option, there needs to be a file specified to keep mpiexec happy, and fsx-mpi was previously unable to create a socket due to permission issues.

I will check to make sure that we need to create the mnt.0 mount-point and if we need to run chef and install.

Here's what appears to be a bunch of problems:

mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt /home/ubuntu/cephtest/fsx-mpi 1B -N 50000 -p 10000 -l 1048576

works when written as:

mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt sudo /home/ubuntu/cephtest/fsx-mpi -o 1B -N 50000 -p 10000 -l 1048576 /home/ubuntu/cephtest/gmnt/test

It appears that there is a missing -o , there needs to be a file specified at the end to keep mpiexec happy, and fsx-mpi was previously unable to create a socket due to permission issues.

The previous mount point issue and chef install issues are not problems.

The two client multisite test is still having issues in that the mpi-hosts file
contains ip-addresses of the host and the client itself. Removing the ip-address
to oneself causes this mpiexec command to work. mpiexec also runs without the mpi-hosts file but then this just runs to oneself instead of across the net (not too exciting for network testing).

Here's the story so far.

The teuthology tests for mpi testing used a yaml file with the following tasks:

# make sure we get the same MPI version on all hosts
os_type: ubuntu
os_version: "14.04" 

tasks:
- pexec:
    clients:
      - cd $TESTDIR
      - wget http://ceph.com/qa/fsx-mpi.c
      - mpicc fsx-mpi.c -o fsx-mpi
      - rm fsx-mpi.c
      - ln -s $TESTDIR/mnt.* $TESTDIR/gmnt
- ssh_keys:
- mpi:
    exec: $TESTDIR/fsx-mpi 1MB -N 50000 -p 10000 -l 1048576
    workdir: $TESTDIR/gmnt
- pexec:
    all:
      - rm $TESTDIR/gmnt
      - rm $TESTDIR/fsx-mpi

There were a few problems with the mpi exec:
1. the syntax was incorrect
2. there should be a file name specified in the mpi exec call
3. there appeared to be permission problems.

So i changed that line to:

exec: sudo $TESTDIR/fsx-mpi -o 1MB -N 50000 -p 10000 -l 1048576 $TESTDIR/gmnt/test

and things worked better.

Next, the clients would not run because mpicc was not installed when the clients ran
that command. So I added a 'sudo apt-get install libmpich2-dev -y' call before the
mpicc command.

The results of one of my runs is in ~ubuntu on teuthology.front.sepia.ceph.com.
This file contains the yaml file that I tested with, the teuthology.log output and a
copy of a session that I later ran on one of the machines.

It looks like things fail but some mpi testing is happening. The mpi-hosts file used in the failing mpiexec statement contains the ip-addresses of the clients.

So when the runs failed, I dropped into interactive mode and ran a session on one of
the clients:

login: Wed Jan  7 02:08:51 2015 from wusui.ubuntu.vpn.sepia.ceph.com
ubuntu@vpm166:~$ ls
cephtest
ubuntu@vpm166:~$ cd cephtest
ubuntu@vpm166:~/cephtest$ ls
archive  data  fsx-mpi  gmnt  mnt.0  monmap  mpi-hosts  valgrind.supp
ubuntu@vpm166:~/cephtest$ ls -ls
total 97
 4 drwxr-xr-x 4 ubuntu ubuntu  4096 Jan  7 02:08 archive
 4 drwxr-xr-x 2 ubuntu ubuntu  4096 Jan  7 02:14 data
72 -rwxrwxr-x 1 ubuntu ubuntu 70847 Jan  7 02:16 fsx-mpi
 0 lrwxrwxrwx 1 ubuntu ubuntu    27 Jan  7 02:16 gmnt -> /home/ubuntu/cephtest/mnt.0
 1 drwxrwxrwt 1 root   root       0 Jan  7 02:16 mnt.0
 4 -rw-rw-r-- 1 ubuntu ubuntu   469 Jan  7 02:14 monmap
 4 -rw-rw-r-- 1 ubuntu ubuntu    43 Jan  7 02:16 mpi-hosts
 8 -rw-r--r-- 1 root   root    4921 Jan  7 02:14 valgrind.supp
ubuntu@vpm166:~/cephtest$ ls mnt.0
test  test.fsxgood  test.fsxlog
ubuntu@vpm166:~/cephtest$ ls -!!
ls -ls mnt.0
total 1025
1024 -rw-r--r-- 1 root root 1048576 Jan  7 02:25 test
   0 -rw-r--r-- 1 root root       0 Jan  7 02:16 test.fsxgood
   1 -rw-r--r-- 1 root root     796 Jan  7 02:25 test.fsxlog
ubuntu@vpm166:~/cephtest$ df mnt.0
Filesystem     1K-blocks    Used Available Use% Mounted on
ceph-fuse      629145600 3637248 625508352   1% /home/ubuntu/cephtest/mnt.0
ubuntu@vpm166:~/cephtest$ mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt sudo /home/ubuntu/cephtest/fsx-mpi -o 1MB -N 50000 -p 10000 -l 1048576 /home/ubuntu/cephtest/gmnt/test
Warning: Permanently added '10.214.139.181' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.214.140.88' (ECDSA) to the list of known hosts.
skipping zero size read
truncating to largest ever: 0x7cccb
truncating to largest ever: 0x974ec
truncating to largest ever: 0xcc932
Size error: expected 0x100000 stat 0x0 seek 0x0
LOG DUMP (25 total operations):
1(1 mod 256): SKIPPED (no operation)
2(2 mod 256): WRITE 0x1c748 thru 0xa6f7b    (0x8a834 bytes) HOLE
3(3 mod 256): WRITE 0x7ea33 thru 0x7fc14    (0x11e2 bytes)
4(4 mod 256): READ  0x21437 thru 0x42e18    (0x219e2 bytes)
5(5 mod 256): MAPREAD   0x6f92f thru 0x77b31    (0x8203 bytes)
6(6 mod 256): WRITE 0x1860 thru 0x8dc6f (0x8c410 bytes)
7(7 mod 256): TRUNCATE DOWN from 0xa6f7c to 0x7cccb
8(8 mod 256): READ  0x2568c thru 0x33f2c    (0xe8a1 bytes)
9(9 mod 256): READ  0x3507c thru 0x7ccca    (0x47c4f bytes)
10(10 mod 256): READ    0x258b1 thru 0x7ccca    (0x5741a bytes)
11(11 mod 256): READ    0x27810 thru 0x2eb87    (0x7378 bytes)
12(12 mod 256): WRITE   0xf46f8 thru 0xfffff    (0xb908 bytes) HOLE
13(13 mod 256): MAPWRITE 0xb407e thru 0xfffff   (0x4bf82 bytes)
14(14 mod 256): TRUNCATE DOWN   from 0x100000 to 0x974ec
15(15 mod 256): MAPWRITE 0xb4b97 thru 0xfffff   (0x4b469 bytes)
16(16 mod 256): READ    0x9ae55 thru 0xfffff    (0x651ab bytes)
17(17 mod 256): READ    0x5d1b1 thru 0x6a8bb    (0xd70b bytes)
18(18 mod 256): READ    0x4b973 thru 0xff34b    (0xb39d9 bytes)
19(19 mod 256): TRUNCATE DOWN   from 0x100000 to 0x46d53
20(20 mod 256): WRITE   0xb4cb3 thru 0xfffff    (0x4b34d bytes) HOLE
21(21 mod 256): MAPWRITE 0x70e38 thru 0xcea78   (0x5dc41 bytes)
22(22 mod 256): TRUNCATE DOWN   from 0x100000 to 0xcc932
23(23 mod 256): MAPREAD 0x9080e thru 0x9456c    (0x3d5f bytes)
24(24 mod 256): TRUNCATE DOWN   from 0xcc932 to 0x3395
25(25 mod 256): MAPWRITE 0xe4b2a thru 0xfffff   (0x1b4d6 bytes)
Correct content saved for comparison
(maybe hexdump "/home/ubuntu/cephtest/gmnt/test" vs "/home/ubuntu/cephtest/gmnt/test.fsxgood")

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 120
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0:2@vpm174] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:0:2@vpm174] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:2@vpm174] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event
skipping zero size read
truncating to largest ever: 0x7cccb
truncating to largest ever: 0x974ec
truncating to largest ever: 0xcc932
truncating to largest ever: 0xe65c0
truncating to largest ever: 0xf9a4e
truncating to largest ever: 0xfa81d
truncating to largest ever: 0xfe94f
truncating to largest ever: 0xfef2b
truncating to largest ever: 0xff5c1
truncating to largest ever: 0xff95c
truncating to largest ever: 0xffb0d
truncating to largest ever: 0xffe56
10000 mapread   0x4c625 thru    0xa99dc (0x5d3b8 bytes)
truncating to largest ever: 0xffed0
20000 mapread   0xb15e5 thru    0xb3378 (0x1d94 bytes)
truncating to largest ever: 0xfff56
30000 trunc from 0xef6dc to 0x2ed5b
40000 write 0xb5eb7 thru    0xfffff (0x4a149 bytes)
50000 mapread   0x1b81a thru    0x46a21 (0x2b208 bytes)
All operations completed A-OK!
ubuntu@vpm166:~/cephtest$ 

It looks like we are testing a ceph-fuse mount. The mpi results both indicate that
one set of operations worked, while another failed.

On previous runs, I experimented with having different entries in the mpi-hosts file, and I also removed files written on mnt.1 to see if that had any affect (I was guessing that the 120 exitcode was an EISNAM that was messing up). After I went down that rabbit-hole for a while, I realized that taking pot-shots was not effective, and
I started to consider the possibility that maybe this test is uncovering a problem
with ceph to begin with. So I decided to post this information and see if anyone could shed light on this.

The machines are vpm156 vpm166 vpm174 and vpm197 and I still have them locked.

changes look ok, although you can probably just use mnt.0 instead of the gmnt symlink.

also, make sure the -dev package is installed in the same way/place as mpi itself. not sure if that is done by the task in teuthology or as part of ceph-qa-chef?

See #10489 for a related issue.

Hmm. I have a branch with this commit: f93e531de6fa69a3ad32117c613094fc0aa0283e
(same as the one mentioned a few updates back), but it does not seem to be on github.

I have been waylaid by QE work. I'll get a wip branch pushed again.