Ceph » CephFS

Hmm, could be that we just have no limit on the number of dirty objects (trim is only dropping clean things), so if we're not writing out to rados quickly enough then the client buffer is just growing and growing.

I see ... so the fix would be to have a dirty_limit (like vm.dirty_ratio) above which all new request would not be cacheable anymore? I guess this doesn't exist yet, does it? Any other way to limit the client-side cache? Or give back-pressure to the client?

Hmm, I've looked at the code now and it turns out we do already have a client_oc_max_dirty setting that is supposed to block writes and wait for writeback, so something else is going on here.

Just in case it wasn't clear from the log snippet above: the current counter is continously growing until the process gets killed. The original (default) value was 1000, I set it to 100 to see if setting it explicitly would make a difference. How can I confirm that the client uses the 100MB default for client_oc_max_dirty and how can I see how much the current amount of dirty bytes?

Hmm.

I've tried setting some comically low limits:

        client oc size = 10485760                                               
        client oc max dirty = 5242880                                           
        client oc max objects = 1   

and found that it is indeed keeping the cache small when I write a large file. The object count does go up to 3/4 rather than 1, so it's clearly not completely strict, but the client_oc_max_dirty setting blocking writes is kicking in.

What are the settings you've got?

I don't think I have changed the defaults. How can I see the values I use?

The log says "objects: max 100" but the default of client_oc_max_objects is 1000, so I assumed there was some manual configuration going on.

If you find the .asok file for the fuse client in /var/lib/ceph, then do a "ceph daemon <asok> config show" you can dump the current configuration. There may or may not be an asok file though depending on permissions and configuration (sometimes I set "admin_socket = /tmp/$cluster-$name.asok" for debugging a client).