Ceph » CephFS

Somehow I got into a situation in which a number of symlinks, all of them created and later modified at about the same time, are reported as having size zero. Here's an example:

$ stat yes
File: ‘yes’ -> ‘../../bin/busybox’
Size: 0 Blocks: 0 IO Block: 4194304 symbolic link
Device: 3eh/62d Inode: 1099514267978 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-03-29 04:21:35.093710964 -0300
Modify: 2011-02-20 16:51:08.000000000 -0300
Change: 2013-03-29 04:21:43.468081000 -0300
Birth: -

Other symlinks report the length of the links-to pathname as their size. The command above was run within a ceph-fuse mountpoint. If I try to run it within a ceph.ko mountpoint (now running 3.8.5, first got with 3.8.4), I get:

$ stat yes
stat: cannot stat ‘yes’: Invalid argument
$ readlink yes
Killed

stracing reveals the stat syscall returns -EINVAL, and the readlink program just dies. Both trigger oopses, one in fill_inode, the other in skip_spaces (kernel dereferences a NULL pointer).

The symlinks were created with rsync in a ceph.ko mountpoint, and they had their timestamps changed at a later time (the one shown in the stat output above) in a ceph-fuse mountpoint. There may have been abrupt ceph-fuse and ceph-mds restarts. Abrupt reboots of the ceph.ko client are also a possibility, but less likely.

I looked a bit in the ceph-osd file holding the directory that contains the symlink, and I can see ^Q in the yes_head directory entry, but I didn't check whether that's just the symlink length, or if it could also be the size stored in the inode. I'll let the filesystem run a while longer so as to get the mds cached/journaled entry for the symlink pushed out and reloaded from the disk. Anyway, I thought I'd report this because this sort of size corruption Shouldn't Happen, and you guys might want to try to look into it and try to find out a way it could have happened. In all the two years I've used cephfs, I'd never seen this before.

Oh, the symlink was likely created when I was running ceph 0.58, or maybe even 0.57. The symlink timestamp update on a ceph-fuse mountpoint mentioned above certainly took place after I'd rolled 0.59 out onto all cluster members and clients.