Ceph » CephFS

Are you sure that the defaults for recalling aren't overly conservative?

Today debugging a situation with 2 heavy clients I came up with this approach: I scale the following options linearly to drive more recall:

X=$1

echo Scaling MDS Recall by ${X}x
ceph tell mds.* injectargs -- --mds_recall_max_decay_threshold $((X*16*1024)) --mds_recall_max_caps $((X*5000)) --mds_recall_global_max_decay_threshold $((X*64*1024)) --mds_recall_warning_threshold $((X*32*1024)) --mds_cache_trim_threshold $((X*64*1024))

The above scales several related options all by X from their defaults. With this, scaling to 8x, I could keep my MDS under 4GB or even 1GB ram used if I needed to, even though the clients were find'ing inodes at more than 30kHz.

Do we really want to throttle clients when the MDS is itself capable of higher performance?

Dan van der Ster wrote:

Are you sure that the defaults for recalling aren't overly conservative?

Yes, the probably are.

Today debugging a situation with 2 heavy clients I came up with this approach: I scale the following options linearly to drive more recall:

[...]

The above scales several related options all by X from their defaults. With this, scaling to 8x, I could keep my MDS under 4GB or even 1GB ram used if I needed to, even though the clients were find'ing inodes at more than 30kHz.

Do we really want to throttle clients when the MDS is itself capable of higher performance?

I think it still makes sense to add a throttle. If a client is too slow releasing caps even with aggressive recall, the MDS should slow down handing out large numbers of caps so the client can keep up.

Is it related to MDS cache overconsumption?

Mykola Dvornik wrote:

Is it related to MDS cache overconsumption?

Yes.