cephfs: clients should send usage metadata to MDSs for administration/monitoring
- The capability "cache hits" by clients to provide introspection on the effectiveness of client capability caching. (Slightly different from #26996, this provides aggregate information for all caps from the client.)
- OSDC cache hits.
- OSDC readahead utilization (?); i.e. if the client reads ahead, does its workload actually use those buffers?
- data write/read ops/bandwidth
In terms of implementation, this will be a new message sent to one of the ranks the client has a session with. It is probably not important which rank it sends the message to or that the same rank gets the sequential messages. Periodically, the ranks will forward these messages in a reduced form to rank 0. Rank 0, in a separate thread, can then aggregate all session metadata and send statistics to the mgr. This will probably include a snapshot of all sessions with various per-session statistics (#24285). (It's probably a good idea to send all of the sessions statistics to the mgr even if that is potentially ~1000 sessions. The user of `ceph fs top` may want to search for a particular session.)