Ceph » CephFS

Dhairya Parmar wrote:

Currently the code to report health warning about laggy clients due to laggy OSDs in mds/Beacon.cc is buggy since it reports:

[...]

because the current code in Beacon.cc checks if the laggy_clients set is non-empty. This is erroneous and must be fixed:
1) if any osd is laggy and there are laggy_clients with defer_client_eviction_on_laggy_osds true: Client X is laggy; not evicted because some OSD is/are laggy
2) if any osd is laggy and there are laggy_clients but defer_client_eviction_on_laggy_osds is unset: Client X is laggy because some OSD is/are laggy

I.e. we will continue reporting clients that are laggy due to laggy osds but we will not say they are evicted when config defer_client_eviction_on_laggy_osds is unset/false/off.

Doesn't the laggy clients list get cleared here: https://github.com/ceph/ceph/blob/main/src/mds/MDSRank.cc#L752 ?

Venky Shankar wrote:

Dhairya Parmar wrote:

Currently the code to report health warning about laggy clients due to laggy OSDs in mds/Beacon.cc is buggy since it reports:

[...]

because the current code in Beacon.cc checks if the laggy_clients set is non-empty. This is erroneous and must be fixed:
1) if any osd is laggy and there are laggy_clients with defer_client_eviction_on_laggy_osds true: Client X is laggy; not evicted because some OSD is/are laggy
2) if any osd is laggy and there are laggy_clients but defer_client_eviction_on_laggy_osds is unset: Client X is laggy because some OSD is/are laggy

I.e. we will continue reporting clients that are laggy due to laggy osds but we will not say they are evicted when config defer_client_eviction_on_laggy_osds is unset/false/off.

Doesn't the laggy clients list get cleared here: https://github.com/ceph/ceph/blob/main/src/mds/MDSRank.cc#L752 ?

this patch is when we try report laggy clients post finding them in find_idle_sessions and evict_cap_revoke_non_responders, check detailed explanation https://github.com/ceph/ceph/pull/53839#issuecomment-1748916319

Hello.

We just upgraded one of our ceph clusters from 16.2.13 to 16.2.14. After the upgrade we have problems with our cephfs when we reboot servers that are mounting those.
The cluster often goes into a state like:

ceph --cluster es1 health detail
HEALTH_WARN 1 client(s) laggy due to laggy OSDs; 1 clients failing to respond to capability release; 1 MDSs report slow requests
[WRN] MDS_CLIENTS_LAGGY: 1 client(s) laggy due to laggy OSDs
    mds.DE-ES-001-03-09-05-2(mds.0): Client 3097440721 is laggy; not evicted because some OSD(s) is/are laggy
[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
    mds.DE-ES-001-03-09-05-2(mds.0): Client DE-ES-001-03-07-01-8:cephfs failing to respond to capability release client_id: 3097440721
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
    mds.DE-ES-001-03-09-05-2(mds.0): 3 slow requests are blocked > 30 secs

We were only able to clear this by restarted the currently active mds. The cluster has no laggy osds at that point anywhere, only a hosting cephfs server was rebooted.

After digging around we found that https://github.com/ceph/ceph/pull/52270 was included in the 16.2.14 release so we disabled `defer_client_eviction_on_laggy_osds` which fixes the issue for now in our env.

Could it be that our problem is related to this bug report here or should we create a new one?

Hi Maximilian,

Maximilian Stinsky wrote:

Hello.

We just upgraded one of our ceph clusters from 16.2.13 to 16.2.14. After the upgrade we have problems with our cephfs when we reboot servers that are mounting those.
The cluster often goes into a state like:

[...]

We were only able to clear this by restarted the currently active mds. The cluster has no laggy osds at that point anywhere, only a hosting cephfs server was rebooted.

After digging around we found that https://github.com/ceph/ceph/pull/52270 was included in the 16.2.14 release so we disabled `defer_client_eviction_on_laggy_osds` which fixes the issue for now in our env.

Could it be that our problem is related to this bug report here or should we create a new one?

Its likely that you are running into this bug. For now, please run with the config disabled.

/a/yuriw-2023-10-25_14:34:26-rados-wip-yuri5-testing-2023-10-24-0737-pacific-distro-default-smithi/7436955