https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2022-11-07T13:48:52ZCeph CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2280262022-11-07T13:48:52ZVenky Shankarvshankar@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Triaged</i></li><li><strong>Assignee</strong> set to <i>Ramana Raja</i></li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2290742022-12-08T05:52:35ZVenky Shankarvshankar@redhat.com
<ul><li><strong>Assignee</strong> changed from <i>Ramana Raja</i> to <i>Venky Shankar</i></li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2300852023-01-12T13:21:22ZVenky Shankarvshankar@redhat.com
<ul></ul><p>Venky Shankar wrote:</p>
<blockquote>
<p><a class="external" href="https://bugzilla.redhat.com/show_bug.cgi?id=2134709">https://bugzilla.redhat.com/show_bug.cgi?id=2134709</a></p>
<p>Generally seen when the MDS is heavily loaded with I/Os. Interesting thing is that the client-id in the warning message is (more often than not) the ceph-mgr daemon (libcephfs instance in ceph-mgr), although the libcephfs instance is <strong>not</strong> the one that is doing heaving I/Os (but there seems to be a possible in-direct relation somehow) - which needs RCA/explanation. Furthermore, it is possible that there are bugs lurking around client tid management which could cause these warnings.</p>
</blockquote>
<p>I've started to look into this - it is not always the ceph-mgr daemon that gets reported in the warning. Maybe it was just coincidence. The tid management in the client and the mds looks sane too. What can be done for better debugability is to dump the oldest_tid that clients maintain. That would give some insights into which request was not acknowledged by the mds.</p> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2302222023-01-17T12:01:19ZVenky Shankarvshankar@redhat.com
<ul></ul><p>partial fix (debug aid): <a class="external" href="https://github.com/ceph/ceph/pull/49766">https://github.com/ceph/ceph/pull/49766</a></p> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2305152023-01-24T12:10:16ZVenky Shankarvshankar@redhat.com
<ul></ul><p>Enforce (stricter) client-id check in client limit test - <a class="external" href="https://github.com/ceph/ceph/pull/49844">https://github.com/ceph/ceph/pull/49844</a></p> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2325432023-03-08T02:39:02ZVenky Shankarvshankar@redhat.com
<ul><li><strong>Status</strong> changed from <i>Triaged</i> to <i>Pending Backport</i></li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2328892023-03-13T16:51:13ZBackport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/59021">Backport #59021</a>: quincy: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads</i> added</li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2328912023-03-13T16:51:24ZBackport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-10 priority-4 priority-default closed" href="/issues/59022">Backport #59022</a>: pacific: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads</i> added</li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2328922023-03-13T16:51:24ZBackport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/59023">Backport #59023</a>: pacific: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads</i> added</li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2328952023-03-13T16:51:29ZBackport Bot
<ul><li><strong>Tags</strong> set to <i>backport_processed</i></li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2328962023-03-13T16:51:37ZBackport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-10 priority-4 priority-default closed" href="/issues/59024">Backport #59024</a>: quincy: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads</i> added</li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2443852023-08-15T08:58:03ZKonstantin Shalygink0ste@k0ste.ru
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2492072023-10-30T01:31:34ZXiubo Lixiubli@redhat.com
<ul></ul><p>Venky, this also should be backport to <strong>reef</strong>, else it caused the failures <a class="external" href="https://tracker.ceph.com/issues/63339">https://tracker.ceph.com/issues/63339</a>.</p> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2492282023-10-30T10:02:37ZVenky Shankarvshankar@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-9 status-2 priority-4 priority-default" href="/issues/63339">Backport #63339</a>: reef: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloads</i> added</li></ul> CephFS - Bug #57985: mds: warning `clients failing to advance oldest client/flush tid` seen with some workloadshttps://tracker.ceph.com/issues/57985?journal_id=2492302023-10-30T10:03:55ZVenky Shankarvshankar@redhat.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Pending Backport</i></li><li><strong>Target version</strong> changed from <i>v18.0.0</i> to <i>v19.0.0</i></li><li><strong>Backport</strong> changed from <i>pacific,quincy</i> to <i>pacific,quincy,reef</i></li></ul>