Ceph » CephFS

Background:

• Ubuntu Server 12.04 64bit.
• CephFS Dumpling 0.67.4
• We moved from local filesystem to CephFS for web data.
• Webserver mounts CephFS on /mnt/ceph
• In order to reduce some CephFS I/O, and without wanting to change code, I've symlinked some directories within /mnt/ceph back to local filesystem.

Problem:

We use PHP's readdir() function within our code, and noticed that it has a problem with symlinks on CephFS.

Given the following situation:

/mnt/ceph/ -- CephFS mount
/mnt/ceph/tenants/0002/data -- Real CephFS directory
/mnt/ceph/tenants/0002/testing -- Symlink to /mnt/ceph/tenants/0002/data

www-data@appserver:/mnt/ceph/tenants/0002$ ls -lha
total 1,5K
drwxr-xr-x 1 www-data www-data 567G Oct 26 07:14 .
drwxr-xr-x 1 www-data www-data 567G Oct 21 18:58 ..
drwxr-xr-x 1 www-data www-data 567G Oct 26 07:10 data
lrwxrwxrwx 1 www-data www-data 4 Oct 26 07:14 testing -> data

#PHP Code:

www-data@appserver:/mnt/ceph/tenants/0002$ cat /var/tmp/test.php

$dirName = $argv[1];
        $dir = opendir($dirName);
        while (($dirEntry = readdir($dir)) !== false) {
                echo $dirEntry . "\n";
        }
?>

The following works:

www-data@appserver:/mnt/ceph/tenants/0002$ php /var/tmp/test.php /mnt/ceph/tenants/0002/data/
.
..
generatedDocument
html
images

But the following results in infinite loop:

www-data@appserver:/mnt/ceph/tenants/0002$ php /var/tmp/test.php /mnt/ceph/tenants/0002/testing/

--- infinite loop ---

I've run the same test successfully on the local file system with symlinks, causing me to suspect CephFS instead of readdir().

I've also tested in C, using readdir(), resulting in failure:

Code: /tmp/readdir.c

#include <stdio.h>
#include <sys/types.h>
#include <dirent.h>

int main ( int argc, char *argv[] ) {
DIR *dp;
struct dirent *ep;
dp = opendir (argv¹);

if (dp != NULL)                                                                                                                                                                   
{                                                                                                                                                                                 
    while (ep = readdir (dp))                                                                                                                                                       
      puts (ep->d_name);                                                                                                                                                            

(void) closedir (dp);                                                                                                                                                           
  }
  else
    perror ("Couldn't open the directory");

return 0;
}

www-data@appserver:/mnt/ceph/tenants/0002$ /tmp/readdir data/
.
..
generatedDocument
html
images

www-data@appserver:/mnt/ceph/tenants/0002$ /tmp/readdir testing/
Couldn't open the directory: Invalid argument

Keep in mind that the above does work with symlinks on local filesystem.

For interest sake, we've found that PHP readdir() on CephFS symlinks will behave if calling is_dir() before readdir():

www-data@appserver:/mnt/ceph/tenants/0002$ cat /var/tmp/workaround.php

$dirName = $argv[1];
        if (is_dir($dirName)) {
                $dir = opendir($dirName);
                while (($dirEntry = readdir($dir)) !== false) {
                        echo $dirEntry . "\n";
                }
        }
?>

www-data@appserver:/mnt/ceph/tenants/0002$ php /var/tmp/workaround.php /mnt/ceph/tenants/0002/testing/
.
..
generatedDocument
html
images