Ceph

under CentOS 6.5, when enable DIO and initialize OSD directory will cause failed:

2014-02-14 05:46:15.591387 7ffff7fe9760 -1 filestore(store_test_temp_dir) mount failed to open journal store_test_temp_journal: (22) Invalid argument

FileJournal open journal with O_DIRECT flag and call posix_fallocate next. The return value will be 22. From implementation of posix_fallocate, it will read 1 byte per block:

  for (offset += (len - 1) % f.f_bsize; len > 0; offset += f.f_bsize)
    {
      len -= f.f_bsize;

      if (offset < st.st_size)
    {
      unsigned char c;
      ssize_t rsize = __pread (fd, &c, 1, offset);

      if (rsize < 0)
        return errno;
      /* If there is a non-zero byte, the block must have been
         allocated already.  */
      else if (rsize == 1 && c != 0)
        continue;
    }

In order to verify, I write a simple program to test under CentOS 6.5:

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdint.h>

int main()
{
    uint64_t d = 400<<20;
    int r;
    int fd = open("dd.tmp", O_CREAT|O_RDWR|O_DIRECT|O_DSYNC, 0644);
    printf("%ld\n", d);
    printf("%d\n", fd);
    r = ::ftruncate(fd, d);
    printf("%d\n", r);
    r = ::posix_fallocate(fd, 0, d);
    printf("%d\n", r);
    close(fd);
    return 0;
}

The result confirm my assumption.

# ./dd
419430400
3
0
22