Ceph » CephFS

FWIW i can trigger the same problems here without creates. `ls -lR` of a large tree is enough, and increasing the throttle limits can help but not in every case.

Dan van der Ster wrote:

FWIW i can trigger the same problems here without creates. `ls -lR` of a large tree is enough, and increasing the throttle limits can help but not in every case.

Yes, that makes sense of course. The problem is not specific to create workloads.

This may be obvious, but to put the whole thing into context: this cache trimming issue can make a CephFS permanently unusable if the MDS hits the physical memory limit. After being kicked by the MON, a standby will continuously try to reload capabilities, fail and be kicked as well.

Janek Bevendorff wrote:

This may be obvious, but to put the whole thing into context: this cache trimming issue can make a CephFS permanently unusable if the MDS hits the physical memory limit. After being kicked by the MON, a standby will continuously try to reload capabilities, fail and be kicked as well.

Yes we also hit that in prod. MDS's were flapping / running out of memory trying to reload the caps.
A solution in our case was to umount cephfs from the relevant clients, so they wouldn't reconnect and their caps wouldn't need to be reloaded the next time the standby MDS tried to start.

Maybe helpful here would be a more conservative cache reservation and an earlier warning on going over the memory limit:

mds cache reservation = 0.2  # default 0.05. reserve more LRU space to handle peak client loads
mds health cache threshold = 1.2  # default 1.5. Early warning if the mds cache is not being trimmed

I also planned to try out the `mds_cap_revoke_eviction_timeout` to see if these clients can be evicted early before breaking the cluster.

We did much the same thing in the OSD. Previously we trimmed in a single thread at regular intervals, but now we trim on add in whatever thread issues the add. We also reduced lock contention (buffer/onode caches shared the same lock) which also helped. That work proved to be very beneficial with a 25-30% increase in 4K random write IOPS in bluestore. The red "wip-bs-cache-evict" numbers are what we gained just with the trim behavior changes:

https://github.com/ceph/ceph/pull/28597

I have the following settings now, which seem to work okay-ish:

mds         advanced mds_beacon_grace                   120.000000                               
mds         basic    mds_cache_memory_limit             26843545600                              
mds         advanced mds_cache_reservation              0.100000                                 
mds         advanced mds_cache_trim_threshold           524288                                   
mds         advanced mds_health_cache_threshold         1.200000                                 
mds         advanced mds_recall_max_caps                15360                                    
mds         advanced mds_recall_max_decay_rate          1.000000

I have three active MDSs and each one is handling about 300-2500 requests/s with a total throughput of 100-300MB/s. The latter is not much and in theory I should be getting up to 2x10Gbps, but considering that this is a latency-dominated task with millions of small files, I guess it's alright.

I believe this problem may be particularly severe when the main data pool is an EC pool. I am trying the same thing with a replicated pool now and am having a lot less issues with this.

As this won't be backported to luminous and many of the mentioned mds options don't exist in luminous, is there a way to address or at least somewhat improve this in luminous?

Jan Fajerski wrote:

As this won't be backported to luminous and many of the mentioned mds options don't exist in luminous, is there a way to address or at least somewhat improve this in luminous?

Luminous is EOL. We'd recommend you upgrade to Mimic or Nautilus.

The other options I spoke about in the mailing list should help mitigate the problem. You can also try setting the mds upkeep thread timeout, mds_tick_interval, to 1.