Ceph » CephFS

Note that the test I ran was on cfuse (most likely because I had kclient trouble, and wanted to isolate that out).

I just cloned the job to http://autotest.ceph.newdream.net/afe/#tab_id=view_job&object_id=555 and that one uses kclient. Results in ~20 minutes.

Job 555 broke, here's a re-run: http://autotest.ceph.newdream.net/afe/#tab_id=view_job&object_id=556

And that confirms that kclient works just fine.

Though it's interesting to see that the "cleanup" phase of dbench took 606 seconds = 10 minutes. On cfuse, I aborted after 744 seconds of "cleanup", after seeing an ls hang. So maybe this is just both being problematic, and cfuse being way slower. I'll rerun cfuse and let it sit for a long time.

kclient dbench log:
http://autotest.ceph.newdream.net/results/556-tv/group0/sepia60.ceph.dreamhost.com/debug/client.0.DEBUG

cfuse dbench log:
http://autotest.ceph.newdream.net/results/554-tv/group0/sepia63.ceph.dreamhost.com/ceph_dbench.cluster0/debug/ceph_dbench.cluster0.DEBUG

My bad, the cleanup phase starts at 600 seconds, so kclient only had a few seconds of cleanup.

The cfuse re-run is at http://autotest.ceph.newdream.net/afe/#tab_id=view_job&object_id=560

Job 560 has spent 1.5 hours in cleanup now, aborting.

14:40:26 DEBUG| [stdout] 2 19 0.00 MB/sec cleanup 6288 sec

http://autotest.ceph.newdream.net/results/560-tv/group0/sepia63.ceph.dreamhost.com/debug/client.0.log

I'm unable to reproduce this on my own machine, and after looking through the mds logs from autotest everything looks good. I'll need a debug log from the client to diagnose this, but it's not clear to me where or how to turn that on with our autotest system.

Okay, hopefully we can rerun this with a time-synced cluster soon and see if that's what is causing the breakage.

I'm seeing a few problems here as well, but not a root cause that would explain what's starting the client down the wrong path in the first place. Most specifically:
It's possible to go into check_caps with an inode popped off the delayed_caps list, then not send the cap update to the MDS. This effectively "loses" the dirtied cap.
There are other things scaring me with those checks. Nothing looks at dirty_caps until you're in the process of sending and while it's possible the checks are valid if all invariants hold, they're obviously fragile.

On the other hand, adding a clock skew option and setting the MDS into the future doesn't let me reproduce the brokenness when running locally.

Scratch that, I did manage to reproduce locally. It just took a bit longer.