Ceph » CephFS

I don't remember where this ended up. Was the proposed fix problematic, or did it never get looked at?

Sage has a different proposed fix than what's in the branch. Still needs to be tested.

#4489 is probably a duplicate of this and has steps to reproduce, if we need alternate angles of attack. (And we should verify that assessment once we fix this bug.)

I'm having difficulty reproducing this at all on current next, but am leaving it churning in the background... :/

I am seeing some other interesting warnings that may need their own bug, though:

INFO:teuthology.orchestra.run.out:access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
INFO:teuthology.orchestra.run.out:------    ---------  ---------- ---------  --------   --------   --------   --------   ----
INFO:teuthology.orchestra.run.out:ior WARNING: inconsistent file size by different tasks.
INFO:teuthology.orchestra.run.out:WARNING: Expected aggregate file size       = 125829120.
INFO:teuthology.orchestra.run.out:WARNING: Stat() of aggregate file size      = 41943040.
INFO:teuthology.orchestra.run.out:WARNING: Using actual aggregate bytes moved = 125829120.
INFO:teuthology.orchestra.run.out:write     200.38     40960      256.00     0.000183   0.598818   0.097357   0.598851   0
INFO:teuthology.orchestra.run.out:ior WARNING: inconsistent file size by different tasks.
INFO:teuthology.orchestra.run.out:WARNING: Expected aggregate file size       = 125829120.
INFO:teuthology.orchestra.run.out:WARNING: Stat() of aggregate file size      = 41943040.

If Locker::_do_cap_update can't get wrlock for a given client, the client should have no Fw cap. I think we can make the clients request new max size only when they have Fw cap. So Locker::_do_cap_update can always take wrlock when handling
new max size request.

• client is writing, and then needs to extend the max size so adds it to caps for flushing
• client receives cap revoke, removing the Fw perm
• client returns cap flush (with the max size request stripped out because it no longer has Fw)
• (time passes and) MDS gives client Fw back
• (if the client had flushed its max size request, it could start writing now, but it hasn't) client requests max size increase
• MDS gives back client max size increase after a round-trip to OSD

And that wasn't working because teuthology was creating working dirs like /tmp/cephtest/gregf@kai-2013-04-16_12-59-21/, and ior wasn't parsing past the '@' sign — no idea why and Sam didn't have one either. However, he tells me newer teuthology won't create dirs that look like that, so I'll just update and we should remember this if we have issues in the future. Hopefully this'll let me reproduce properly!

Reproduced at last. There continues to be a problem with the fix branch too :( but it's not a max_size issue; one of the nodes is failing to get Fr. I'm looking at it now.

Okay, it's not quite that simple. This (all following the data writeout; I think this is the data check — anyway, this) node has previously held pAsLsXsFscr.
Then, the MDS revokes Fsc and the client appears to handle that — _release(), _invalidate_inode_cache(), check_caps() and send_cap(). However, it's currently got Fcr marked as "used", so it...doesn't actually drop Fc. I admit I don't quite remember how this part is supposed to work, but the MDS sure seems to think that at this point the client no longer has those caps (but there might just not be any output of that).
Then later on the MDS does give the client Fr, which is enough for the client to do what it wants, but the client still says that it's got Fsc implemented (so it hasn't cleaned that up yet?), and so when it ends up in get_caps() it notices that it has what it needs, but (revoking & butnot) is non-zero (revoking still containing Fc since it has yet to drop the damn cap, as does butnot) so it continues to wait instead of breaking out.
I'll discuss this with Sage and move forward.

Greg Farnum wrote:

I don't remember how all the locking works when you have multiple writers, but I don't believe either of those suppositions are correct:
1) Multiple clients can have Fw cap at a time (without buffering) during sync writes, but that doesn't mean they can get wrlock immediately. That's part of what the distinction is there for, isn't it?
2) If we do have clients gaining and losing Fw cap while they've got a file opened for write, we'd have to be able to intelligently add and remove their max size change requests on their cap acknowledgements. Besides being more logic in the client, this would require more message traffic before a client can do writes. Eg:

• client is writing, and then needs to extend the max size so adds it to caps for flushing
• client receives cap revoke, removing the Fw perm
• client returns cap flush (with the max size request stripped out because it no longer has Fw)
• (time passes and) MDS gives client Fw back
• (if the client had flushed its max size request, it could start writing now, but it hasn't) client requests max size increase
• MDS gives back client max size increase after a round-trip to OSD

thanks for clarification

Greg Farnum wrote:

Okay, it's not quite that simple. This (all following the data writeout; I think this is the data check — anyway, this) node has previously held pAsLsXsFscr.
Then, the MDS revokes Fsc and the client appears to handle that — _release(), _invalidate_inode_cache(), check_caps() and send_cap(). However, it's currently got Fcr marked as "used", so it...doesn't actually drop Fc. I admit I don't quite remember how this part is supposed to work, but the MDS sure seems to think that at this point the client no longer has those caps (but there might just not be any output of that).

This sounds like a bug I recently fixed, see [PATCH 1/2] mds: pass proper mask to CInode::get_caps_issued.

Greg Farnum wrote:

I don't remember how all the locking works when you have multiple writers, but I don't believe either of those suppositions are correct:
1) Multiple clients can have Fw cap at a time (without buffering) during sync writes, but that doesn't mean they can get wrlock immediately. That's part of what the distinction is there for, isn't it?

there are only 4 states that allow Fw caps, they are MIX, MIX_EXCL, EXCL and EXCL_MIX. they all allow Fw cap.

Greg Farnum wrote:

2) If we do have clients gaining and losing Fw cap while they've got a file opened for write, we'd have to be able to intelligently add and remove their max size change requests on their cap acknowledgements. Besides being more logic in the client, this would require more message traffic before a client can do writes. Eg:

I thought about this again. I think we can allow force wrlock when Fw caps are revoking. (allow force wrlock for EXCL_XSYN, EXCL_SYNC, EXCL_LOCK MIX_SYNC, MIX_LOCK)

Zheng Yan wrote:

there are only 4 states that allow Fw caps, they are MIX, MIX_EXCL, EXCL and EXCL_MIX. they all allow Fw cap.

Sorry, I mean they all allow wrlock

That latest issue was #4746. Turning off the callback and testing again...