Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads - CephFS - Ceph

Actions

Copy link

Bug #57985

open

mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads

Added by Venky Shankar over 1 year ago. Updated 6 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Venky Shankar

Category:

Correctness/Safety

Target version:

Ceph - v19.0.0

% Done:

100%

Source:

Q/A

Tags:

backport_processed

Backport:

pacific,quincy,reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Client, MDS

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

https://bugzilla.redhat.com/show_bug.cgi?id=2134709

Generally seen when the MDS is heavily loaded with I/Os. Interesting thing is that the client-id in the warning message is (more often than not) the ceph-mgr daemon (libcephfs instance in ceph-mgr), although the libcephfs instance is not the one that is doing heaving I/Os (but there seems to be a possible in-direct relation somehow) - which needs RCA/explanation. Furthermore, it is possible that there are bugs lurking around client tid management which could cause these warnings.

Related issues 5 (1 open — 4 closed)

Related to CephFS - Backport #63339: reef: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads	In Progress	Xiubo Li	Actions
Copied to CephFS - Backport #59021: quincy: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads	Resolved	Venky Shankar	Actions
Copied to CephFS - Backport #59022: pacific: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads	Duplicate	Venky Shankar	Actions
Copied to CephFS - Backport #59023: pacific: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads	Resolved	Venky Shankar	Actions
Copied to CephFS - Backport #59024: quincy: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads	Duplicate	Venky Shankar	Actions

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Status changed from New to Triaged
Assignee set to Ramana Raja

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Assignee changed from Ramana Raja to Venky Shankar

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Venky Shankar wrote:

https://bugzilla.redhat.com/show_bug.cgi?id=2134709

Generally seen when the MDS is heavily loaded with I/Os. Interesting thing is that the client-id in the warning message is (more often than not) the ceph-mgr daemon (libcephfs instance in ceph-mgr), although the libcephfs instance is not the one that is doing heaving I/Os (but there seems to be a possible in-direct relation somehow) - which needs RCA/explanation. Furthermore, it is possible that there are bugs lurking around client tid management which could cause these warnings.

I've started to look into this - it is not always the ceph-mgr daemon that gets reported in the warning. Maybe it was just coincidence. The tid management in the client and the mds looks sane too. What can be done for better debugability is to dump the oldest_tid that clients maintain. That would give some insights into which request was not acknowledged by the mds.

Actions

Copy link