Ceph

I haven't fully determined everything going on here, but were you using anything besides the kernel client with this cluster?
Each OSD log you provided has at least one operation that it says is too large to fit into the journal. We've actually been working on better reporting errors like that back to the client, but it's the most problematic thing I've noticed so far that looks like it could be causing the problem.

Also the OSDs aren't managing to peer their PGs, but that's going to be a symptom of something else (possibly this journal-too-small problem, haven't traced it all the way out yet).

Only the kernel client was in use, so am assuming it was the mds that made a write bigger than the journal size (120meg). I've wiped out the cluster and am trying to reproduce.

I can reproduce this:

• format new cluster (in my case: 3 mons, 2 mds, 4 osds)
• mount filesystem using kernel client (2.6.37 vanilla)
• mkdir files
• cd files
• ruby -e '1000000.times { |i| s = i.to_s(24) ; open(s, "w") { |f| puts s ; f.write(s * 1024) } }'

that last line attempts to create a million little files. Along the way, the active mds seems to restart (I'm assuming a crash) and the other mds takes over, but at this time a large write is made to two of the osds which fills the journal (my journal size is 300meg).

the cmds quickly grows to 2gig ram size btw.

Will attach some debug logs if I can get them.

Do you have any core dumps from the MDS? Or is this running in a memory-constrained system so it's getting OOM-killed?

to be clear, this is not resulting in the "tid timed out" messages now, just the journal hangs, so it might be a separate issue. The client kernel blocks until the mds fails over and work continues (suppose it might get to the timed out messages once all the osds are stuck?)

no ooms (the boxes are 6gig ram) and now I'm relatively sure the mds is not crashing, it seems to be an orderly respawn (looks like it might be getting blacklisted for being laggy?)

.....oh. So you're trying to create a million files in a single directory?

Ceph stores inodes in the directory they belong to. I don't know exactly how much on-disk space each inode takes, but in-memory they take ~2KB. Assuming the numbers are similar, a 300MB journal can handle a directory of ~153k inodes.
So, I think what's happening is this:
1) Client creates lots and lots of files in directory.
2) MDS keeps up for a while, but eventually tries to flush out a directory that won't fit into the OSD journal
3) OSD journal fails somehow. (In recent code it should return an error code, but in older code I think it fails silently)
4) MDS either is running slowly, or gets stuck waiting for ack from OSD and eventually gets killed by mon. (This could be due to a blocking behavior that shouldn't exist, actually! But a separate issue and Sage says not likely.)
5) MDS gets marked down and respawns itself.
6) Loop as kernel client replays with MDS, MDS tries to flush out to OSDs, etc etc etc

Something like this anyway! Sage says that this won't be an issue once we get directory fragmentation stable (large directories will fragment into multiple pieces that are committed independently, etc), and this will be done by 1.0, but at the moment there's not a lot to be done about it, except limit the size of your directories to something that fits within your journal. :(

Sage Weil wrote:

And after thinking about it a bit, Greg, _commit_full should be pretty easily fixed up to behave even without fragmentation just by doing multiple writes with some max size in series. I'm pretty sure it doesn't have to be a single atomic commit because of the way the dentry/inode versioning works. Let's look at it tomorrow!

All right, this should be handled in 3129581e89cb38f9ad2ff85673e50a033ec77d40 which is in the master branch. Let me know how it goes. :)

Tested with a build of master and it looks good. I'm writing files to a dir atm and am up to 430,000 entries and still going.