https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-02-06T05:12:42ZCeph CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1069122018-02-06T05:12:42ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>CephFS</i></li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1070102018-02-06T21:06:37ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Subject</strong> changed from <i>LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same file</i> to <i>mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same file</i></li><li><strong>Category</strong> set to <i>Correctness/Safety</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Source</strong> set to <i>Community (user)</i></li><li><strong>Component(FS)</strong> <i>Client, MDS</i> added</li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1071192018-02-08T07:07:08ZXuehan Xuxxhdx1985126@gmail.com
<ul></ul><p>Hi, Patrick, as discussed yesterday, in our case, the whole procedure of a single run of "getattr" ops processing is this:</p>
<ol>
<li>The "finish" phase of the previous "getattr" op processing turn the target inode's filelock state into LOCK_SYNC_MIX if its current state is LOCK_SYNC;</li>
<li>mds blocks all other "getattr" operations when the target inode's filelock state is LOCK_SYNC_MIX, and wait for all clients to release caps that are not allowed in LOCK_MIX;</li>
<li>When all clients release those not allowed caps, the inode's filelock state is turned in to LOCK_MIX;</li>
<li>A new "getattr" op is dispatched to be processed, during which the inode's filelock state is turned back to LOCK_SYNC and the "getattr" op is processed successfully;</li>
</ol>
<p>If the next op that is to be dispatched to be processed is, again, a "getattr", the whole procedure would be repeated. In our case, the waiting of the mds for all clients to release caps that are not allowed in state LOCK_MIX could take several hundreds of milliseconds, which means in one second, only a few "getattr" requests can be processed, and because the "getattr" operation is the mostly issued requests, the above procedure can repeat so many times that the response time of the process of a single "getattr" request can take 40+ seconds as they have to wait for their previous "getattr" ops to be processed first, and in the meantime, the "setattr" operations that are issued by the writing client can hard get the chance to be processed.</p> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1071242018-02-08T10:24:26ZZheng Yanukernel@gmail.com
<ul></ul><p>please try<br /><a class="external" href="https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4">https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4</a></p> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1071812018-02-09T04:35:01ZXuehan Xuxxhdx1985126@gmail.com
<ul></ul><p>Zheng Yan wrote:</p>
<blockquote>
<p>please try<br /><a class="external" href="https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4">https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4</a></p>
</blockquote>
<p>This patch works! Thank you:-)</p> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1072042018-02-09T08:01:22ZXuehan Xuxxhdx1985126@gmail.com
<ul></ul><p>Zheng Yan wrote:</p>
<blockquote>
<p>please try<br /><a class="external" href="https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4">https://github.com/ukernel/ceph/commit/7db1563416b5559310dbbc834795b83a4ccdaab4</a></p>
</blockquote>
<p>Could you make a PR of this commit?<br />We really want to use this commit, however, without the approval of<br />the community, we won't have the courage to put this into our online<br />clusters.<br />Thank you:-)</p> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1072162018-02-09T19:54:39ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Backport</strong> set to <i>luminous,jewel</i></li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1073532018-02-13T19:37:33ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Pending Backport</i></li><li><strong>Assignee</strong> set to <i>Zheng Yan</i></li><li><strong>Component(FS)</strong> deleted (<del><i>Client</i></del>)</li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1073542018-02-13T19:37:58ZPatrick Donnellypdonnell@redhat.com
<ul></ul><p>PR: <a class="external" href="https://github.com/ceph/ceph/pull/20386">https://github.com/ceph/ceph/pull/20386</a></p> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1074252018-02-15T15:15:54ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/23013">Backport #23013</a>: luminous: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same file</i> added</li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1074272018-02-15T15:15:56ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-6 priority-4 priority-default closed" href="/issues/23014">Backport #23014</a>: jewel: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same file</i> added</li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=1231212018-10-18T21:15:19ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul> CephFS - Bug #22925: mds: LOCK_SYNC_MIX state makes "getattr" operations extremely slow when there are lots of clients issue writes or reads to the same filehttps://tracker.ceph.com/issues/22925?journal_id=2424792023-07-18T12:38:37ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-14 priority-4 priority-default" href="/issues/62052">Bug #62052</a>: mds: deadlock when getattr changes inode lockset</i> added</li></ul>