https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2019-05-23T16:00:27Z
Ceph
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=137207
2019-05-23T16:00:27Z
Yunzhi Cheng
wingerted@gmail.com
<ul></ul><p>I set debug_mds to 20/20 and almost all of the log is like</p>
<pre><code class="c syntaxhl"><span class="CodeRay"><span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched pos <span class="integer">217</span> marker <span class="char">'I'</span> dname <span class="char">'a</span>bcd.ab.csv.gz [<span class="integer">2</span>,head]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup (head, <span class="char">'a</span>bcd.ab.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) miss -> (a.T.quote.csv.gz,head)
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup_exact_snap (head, <span class="char">'a</span>bcd.ab.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) add_primary_dentry [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/abcd.ab.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f9b</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b72c0</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched got [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/abcd.ab.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f9b</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b72c0</span>] [inode <span class="hex">0x300001f5f9b</span> [<span class="integer">2</span>,head] /aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/abcd.ab.csv.gz auth v42918 s=<span class="integer">17078</span> n(v0 rc2019-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">13</span>:<span class="integer">22</span>:<span class="integer">5</span><span class="float">3</span><span class="float">.975211</span> b17078 <span class="integer">1</span>=<span class="integer">1</span>+<span class="integer">0</span>) (iversion lock) <span class="hex">0x1431ba700</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched pos <span class="integer">216</span> marker <span class="char">'I'</span> dname <span class="char">'b</span>bbb.bb.csv.gz [<span class="integer">2</span>,head]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup (head, <span class="char">'b</span>bbb.bb.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) miss -> (b.T.quote.csv.gz,head)
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup_exact_snap (head, <span class="char">'a</span>bcd.KQ.others.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) add_primary_dentry [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/bbbb.bb.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f8f</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b74a0</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched got [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/bbbb.bb.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f8f</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b74a0</span>] [inode <span class="hex">0x300001f5f8f</span> [<span class="integer">2</span>,head] /aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/bbbb.bb.csv.gz auth v42102 s=<span class="integer">1878</span> n(v0 rc2019-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">13</span>:<span class="integer">22</span>:<span class="integer">5</span><span class="float">3</span><span class="float">.219209</span> b1878 <span class="integer">1</span>=<span class="integer">1</span>+<span class="integer">0</span>) (iversion lock) <span class="hex">0x1431bae00</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched pos <span class="integer">215</span> marker <span class="char">'I'</span> dname <span class="char">'x</span>xxx.xx.corr.csv.gz [<span class="integer">2</span>,head]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup (head, <span class="char">'x</span>xxx.xx.corr.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) miss -> (c.KS.corr.csv.gz,head)
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup_exact_snap (head, <span class="char">'a</span>bcd.KS.corr.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) add_primary_dentry [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/xxxx.xx.corr.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f95</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b7680</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">12</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched got [dentry <span class="preprocessor">#0x1</span>/aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/xxxx.xx.corr.csv.gz [<span class="integer">2</span>,head] auth (dversion lock) pv=<span class="integer">0</span> v=<span class="integer">178232</span> ino=<span class="hex">0x300001f5f95</span> state=<span class="integer">1073741824</span> <span class="hex">0x1431b7680</span>] [inode <span class="hex">0x300001f5f95</span> [<span class="integer">2</span>,head] /aaa/bbb/ccc/ddd/eee/fff/<span class="integer">2016</span>/<span class="integer">12</span>/<span class="octal">07</span>/xxxx.xx.corr.csv.gz auth v25438 s=<span class="integer">200</span> n(v0 rc2019-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">13</span>:<span class="integer">22</span>:<span class="integer">5</span><span class="float">3</span><span class="float">.227209</span> b200 <span class="integer">1</span>=<span class="integer">1</span>+<span class="integer">0</span>) (iversion lock) <span class="hex">0x1431bb500</span>]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) _fetched pos <span class="integer">214</span> marker <span class="char">'I'</span> dname <span class="char">'a</span>bcd.KS.csv.gz [<span class="integer">2</span>,head]
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup (head, <span class="char">'a</span>bcd.KS.csv.gz<span class="char">')</span>
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) miss -> (d.T.auct.csv.gz,head)
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">22</span> <span class="integer">23</span>:<span class="integer">32</span>:<span class="integer">1</span><span class="float">3</span><span class="float">.588</span> <span class="float">7f</span><span class="integer">4</span>c9f624700 <span class="integer">20</span> mds<span class="float">.0</span>.cache.dir(<span class="hex">0x100012566e9</span><span class="float">.010</span>*) lookup_exact_snap (head, <span class="char">'x</span>xxx.xx.csv.gz<span class="char">')</span>
</span></code></pre>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=137485
2019-05-29T21:47:09Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> changed from <i>v14.2.1</i> to <i>v15.0.0</i></li><li><strong>Start date</strong> deleted (<del><i>05/22/2019</i></del>)</li><li><strong>Backport</strong> set to <i>nautilus</i></li><li><strong>ceph-qa-suite</strong> deleted (<del><i>fs</i></del>)</li><li><strong>Labels (FS)</strong> <i>multimds</i> added</li></ul><p>Are you using snapshots? Can you tell us more about how the cluster is being used like # of clients and versions.</p>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=137502
2019-05-30T05:16:17Z
Yunzhi Cheng
wingerted@gmail.com
<ul></ul><p>Patrick Donnelly wrote:</p>
<blockquote>
<p>Are you using snapshots? Can you tell us more about how the cluster is being used like # of clients and versions.</p>
</blockquote>
<p>I'm not using snapshots.</p>
<p>ceph -s:</p>
<pre><code>cluster:<br /> id: f41c780b-a413-4db5-8bc3-2cd7e81bc275<br /> health: HEALTH_OK</code></pre>
<pre><code>services:<br /> mon: 3 daemons, quorum rndcl94,rndcl106,rndcl154 (age 3d)<br /> mgr: rndcl154(active, since 8d), standbys: rndcl106, rndcl94<br /> mds: cephfs:3 {0=rndcl94=up:active,1=rndcl118=up:active,2=rndcl154=up:active} 1 up:standby<br /> osd: 24 osds: 24 up (since 7d), 24 in (since 7d)</code></pre>
<pre><code>data:<br /> pools: 3 pools, 385 pgs<br /> objects: 123.24M objects, 1.8 TiB<br /> usage: 27 TiB used, 47 TiB / 75 TiB avail<br /> pgs: 381 active+clean<br /> 4 active+clean+scrubbing+deep</code></pre>
<p>All the client are kernel client and kernel version is 4.14.35-041435</p>
<p>Client System is Ubuntu 14.04 and Server System is Ubuntu 16.04</p>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=137503
2019-05-30T08:23:35Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>please if these dirfrag fetches are from open_file_table</p>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=138140
2019-06-08T15:57:42Z
Yunzhi Cheng
wingerted@gmail.com
<ul></ul><p>Zheng Yan wrote:</p>
<blockquote>
<p>please if these dirfrag fetches are from open_file_table</p>
</blockquote>
<p>How can I figure out if they are from open_file_table?</p>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=156127
2020-01-17T22:53:48Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Target version</strong> deleted (<del><i>v15.0.0</i></del>)</li></ul>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=163768
2020-04-22T12:17:27Z
Milind Changire
<ul></ul><p>Yunzhi,<br />What is the value of the config option 'mds_cache_memory_limit' on the system ?<br />Are you referring to this option when referring to MDS being unser 30G memory limit ?</p>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=163832
2020-04-23T04:11:14Z
Milind Changire
<ul><li><strong>Assignee</strong> set to <i>Milind Changire</i></li></ul>
CephFS - Bug #40001: mds cache oversize after restart
https://tracker.ceph.com/issues/40001?journal_id=165348
2020-05-10T04:12:26Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Rejected</i></li></ul><p>This ticket has become stale. Closing.</p>