The symptom is that we're getting ENOSPC from write() calls during buffered IO, where we should be getting them from fclose() calls. I think I've also seen this fail recently on the equivalent fsync() test in TestFull.
We're only seeing this on openstack.
I think the key event is that the client is getting an updated OSDMap when it the test doesn't expect it to. The test does this:
A write(enough data to trigger full flag)
B fsync() <- should succeed because of the delay between writing data and the full flag getting set
C sleep(30) <- full flag should get set during this period, but the client shouldn't find out because it doesn't subscribe continuously to the map
D write(a little more data) <- should succeed because buffered
E fclose() <- should fail because when flushing buffers we'll get the new map and realise that full flag is set.
So during C, we are getting an OSDMap, which is causing D to fail because the client now knows that the full flag is set.