Project

General

Profile

Bug #4601

symlink with size zero

Added by Alexandre Oliva almost 11 years ago. Updated over 10 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Somehow I got into a situation in which a number of symlinks, all of them created and later modified at about the same time, are reported as having size zero. Here's an example:

$ stat yes
File: ‘yes’ -> ‘../../bin/busybox’
Size: 0 Blocks: 0 IO Block: 4194304 symbolic link
Device: 3eh/62d Inode: 1099514267978 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-03-29 04:21:35.093710964 -0300
Modify: 2011-02-20 16:51:08.000000000 -0300
Change: 2013-03-29 04:21:43.468081000 -0300
Birth: -

Other symlinks report the length of the links-to pathname as their size. The command above was run within a ceph-fuse mountpoint. If I try to run it within a ceph.ko mountpoint (now running 3.8.5, first got with 3.8.4), I get:

$ stat yes
stat: cannot stat ‘yes’: Invalid argument
$ readlink yes
Killed

stracing reveals the stat syscall returns -EINVAL, and the readlink program just dies. Both trigger oopses, one in fill_inode, the other in skip_spaces (kernel dereferences a NULL pointer).

The symlinks were created with rsync in a ceph.ko mountpoint, and they had their timestamps changed at a later time (the one shown in the stat output above) in a ceph-fuse mountpoint. There may have been abrupt ceph-fuse and ceph-mds restarts. Abrupt reboots of the ceph.ko client are also a possibility, but less likely.

I looked a bit in the ceph-osd file holding the directory that contains the symlink, and I can see ^Q in the yes_head directory entry, but I didn't check whether that's just the symlink length, or if it could also be the size stored in the inode. I'll let the filesystem run a while longer so as to get the mds cached/journaled entry for the symlink pushed out and reloaded from the disk. Anyway, I thought I'd report this because this sort of size corruption Shouldn't Happen, and you guys might want to try to look into it and try to find out a way it could have happened. In all the two years I've used cephfs, I'd never seen this before.

Oh, the symlink was likely created when I was running ceph 0.58, or maybe even 0.57. The symlink timestamp update on a ceph-fuse mountpoint mentioned above certainly took place after I'd rolled 0.59 out onto all cluster members and clients.

History

#1 Updated by Sage Weil almost 11 years ago

  • Project changed from Ceph to CephFS

#2 Updated by Sage Weil almost 11 years ago

  • Priority changed from Normal to High

#3 Updated by Greg Farnum almost 11 years ago

I looked a bit in the ceph-osd file holding the directory that contains the symlink, and I can see ^Q in the yes_head directory entry, but I didn't check whether that's just the symlink length, or if it could also be the size stored in the inode. I'll let the filesystem run a while longer so as to get the mds cached/journaled entry for the symlink pushed out and reloaded from the disk.

Can you clarify what you were looking at there? Any follow-on updates?

#4 Updated by Alexandre Oliva almost 11 years ago

I was looking at the <inode>.<frag>_head* file in the osd that held the directory where the link was stored. As it turned out, the ^Q was just the length of the string symlinked to, whereas the file size in the inode was indeed zero, so even after flushing from the mds cache and reloading from disk, the inode remained invalid. I ended up removing and re-creating the symlinks.

Another thing I noticed was that an older snapshots of an osd holding that same symlink did NOT have it corrupted, so I'm thinking it was broken either when its timestamp was adjusted with ceph-fuse, or when hardlinks of the symlink were created with ceph.ko. Sorry, I realize this probably doesn't help much, but it's the best info I could collect after-the-fact.

#5 Updated by Sage Weil over 10 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF