Ceph » CephFS

Patrick Donnelly wrote:

Either cephfs-top should handle the missing metadata entries or the mgr/stats should fill in defaults until it can update the metadata.

My intuition is that cephfs-top should be tolerant of the missing metadata entries. Venky, what do you think?

Right. That's one part of the problem that needs to be handled in cephfs-top (or mgr/stats). mgr/stats sets minimum metadata (IP addr) when it sees a client for the first time. Then it sends a (async) request to the MDS for other metadata. The perf stats data handed out by mgr/stats would contain incomplete metadata for this client (since mgr/stats hasn't received the metadata request reply from the MDS).

The other issue which I see in Jos's setup is that the metadata request reply ("client ls" call to the MDS) didn't contain metadata for the client (client.14585):

2021-03-11T09:51:18.622+0000 7fa3252ad700  0 [stats DEBUG root] notify: client metadata=[{'id': 14507, 'entity': {'name': {'type': 'client', 'num': 14507}, 'addr': {'type': 'any', 'addr': '127.0.0.1:0', 'nonce': 1132974732}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 191.8276
00803, 'requests_in_flight': 0, 'completed_requests': [], 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.6421762966800849, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.14507 127.0.0.1:0/1132974732', 'prealloc_inos': [], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x000000000000001f'}}, 'ceph_sha1': '68142daf25e396d4bd8c9caee31c4f0bfe88164f', 'ceph_version': 'ceph version 17.0.0-1725-g68142daf25 (68142daf25e396d4bd8c9caee31c4f0bfe88164f) quincy (dev)', 'entity_id': 'admin', 'hostname': 'smithi069', 'mount_point': '/mnt/cephfs2', 'pid': '22511', 'root': '/'}}, {'id': 14504, 'entity': {'name': {'type': 'client', 'num': 14504}, 'addr': {'type': 'any', 'addr': '127.0.0.1:0', 'nonce': 3460583018}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 192.483585759, 'requests_in_flight': 0, 'completed_requests': [], 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.6418025110207033, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.14504 127.0.0.1:0/3460583018', 'prealloc_inos': [], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x000000000000001f'}}, 'ceph_sha1': '68142daf25e396d4bd8c9caee31c4f0bfe88164f', 'ceph_version': 'ceph version 17.0.0-1725-g68142daf25 (68142daf25e396d4bd8c9caee31c4f0bfe88164f) quincy (dev)', 'entity_id': 'admin', 'hostname': 'smithi069', 'mount_point': '/mnt/cephfs', 'pid': '22470', 'root': '/'}}]

Why did the metadata request did not carry entry for client.14585 needs to be examined further. MDSRank::dump_sessions() has this filter:

if (!filter.match(*s, std::bind(&Server::waiting_for_reconnect, server, std::placeholders::_1))) {
  continue;
}

... which might be the reason that the client got filtered out of the session dump. Jos, could you please check if that's the case?

Venky Shankar wrote:
MDSRank::dump_sessions() has this filter:

[...]

... which might be the reason that the client got filtered out of the session dump. Jos, could you please check if that's the case?

I've checked the mds logs. It doesn't hit there. As of now, there's nothing interesting found for this particular client (client.14585) on the mds side - looks the same as the other clients. Needs more checking.

https://github.com/ceph/ceph/pull/40210