Ceph » CephFS

Hi,
I've been testing the locking feature on cephfs and seems that the locking doesn't work as expected when locking/writing on the same file from 2 different client. I've been testing this on version 0.48

Here are the steps to reproduce the problem.

3 servers used: ceph01, ceph02, ceph03 running each mon and osd + ceph01 running mds
ceph01,ceph02 mount cephfs as follow: (in fstab)
ceph01,ceph02:/ /mnt/ceph ceph name=admin,secretfile=/etc/ceph/secret,noexec,nodev,noatime 0 2

the code used to reproduce the problem:

#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <err.h>

int
main(int argc, char **argv) {
int fd;
char *buf;
struct flock fl;
int count, i;

if (argc < 4) {
        printf("Usage: %s &lt;file&gt; &lt;count&gt; &lt;str&gt;\n", argv[0]);
        exit(1);
    }

if ((fd = open(argv[1], O_RDWR | O_CREAT | O_APPEND, 0644)) == -1)
        err(1, "open");

count = atoi(argv[2]);

asprintf(&buf, "%s\n",  argv[3]);
    if (buf == NULL)
        err(1, "asprintf");

fl.l_type = F_WRLCK;
    fl.l_whence = SEEK_SET;
    fl.l_start = 0;
    fl.l_len = 0;
    fl.l_pid = 0;
    if (fcntl(fd, F_SETLKW, &fl)  -1)
        err(1, "fcntl");
    for (i = 0; i < count || count  0; i++) {
        if (write(fd, buf, strlen(buf)) == -1)
            err(1, "write");
        printf("%d - %s", i + 1, buf);
        sleep(1);
    }
    close(fd);
    return 0;
}

when it works:
1- on the local fs not using local file system (ext4)
steps: on ceph01 open 2 ssh shells and run the following commands:

first shell: ceph01$ ./mr_lock /home/myuser/bla 3 foo; ./mr_lock /home/myuser/bla 3 foo
second shell: ceph01$ ./mr_lock /home/myuser/bla 3 bar; ./mr_lock /home/myuser/bla 3 bar

result: cat bla
foo
foo
foo
bar
bar
bar
foo
foo
foo
bar
bar
bar

2- on the same client using ceph file system
steps: open 2 shells on ceph01 and run the following commands:

first shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 foo; ./mr_lock /mnt/ceph/bla 3 foo
second shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 bar; ./mr_lock /mnt/ceph/bla 3 bar

result: cat /mnt/ceph/bla
foo
foo
foo
bar
bar
bar
foo
foo
foo
bar
bar
bar

when it doesn't work:
1- when 2 clients are connected to ceph

steps: open 1 shell on each ceph01 and ceph02 and run the following commands:

first shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 foo; ./mr_lock /mnt/ceph/bla 3 foo
second shell: ceph02$ ./mr_lock /mnt/ceph/bla 3 bar; ./mr_lock /mnt/ceph/bla 3 bar

result:

ceph01$ cat /mnt/ceph/bla
bar
bar
bar
bar
bar
bar

another try shows this result:

cat /mnt/ceph/bla
foo
bar
bar
foo
bar
bar
bar

Let me know if you need anything additionnal info.