https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2017-01-16T12:15:10ZCeph CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=842212017-01-16T12:15:10ZJohn Sprayjcspray@gmail.com
<ul><li><strong>Subject</strong> changed from <i>subdir with corrupted dirstat is un-rm-able</i> to <i>Forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)</i></li><li><strong>Category</strong> set to <i>fsck/damage handling</i></li><li><strong>Target version</strong> set to <i>v12.0.0</i></li><li><strong>Backport</strong> set to <i>jewel kraken</i></li><li><strong>Component(FS)</strong> <i>MDS</i> added</li></ul><p>Without having looked into this in detail yet, my presumption would be that the bug is that the repair code isn't fixing the stats -- I think the refusal to delete is probably not a bug in itself.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=843342017-01-17T15:13:12ZZheng Yanukernel@gmail.com
<ul></ul><p>maybe there is a bad remote link in the directory</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=848012017-01-25T22:11:49ZDan Mickdmick@redhat.com
<ul></ul><p>I don't know how to repair this or even identify other instances.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=848452017-01-26T07:06:22ZZheng Yanukernel@gmail.com
<ul></ul><p>Fixed by:</p>
<p>ceph daemon mds.mira049 scrub_path /teuthology-archive/sage-2016-11-12_02:26:45-rados-wip-sage-testing---basic-smithi/541839/remote/smithi059/log repair recursive force</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=848462017-01-26T07:13:47ZZheng Yanukernel@gmail.com
<ul></ul><p>"ceph daemon mds.mira049 scrub_path / repair recursive force" will find and fix any other issue. But it will take very log time, I don't know if it‘s worth the effort.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=848542017-01-26T09:23:23ZZheng Yanukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>4</i></li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=849102017-01-27T01:00:39ZDan Mickdmick@redhat.com
<ul></ul><p>I would have sworn Greg directed me to try that, but perhaps we didn't include 'force'. Shrug. Thanks for the help.</p>
<p>I have a multi-day 'find' command looking for huge directories that I'll let continue; it's found at few more so far:</p>
<p>teuthology-2016-12-18_02:01:14-rbd-master-distro-basic-smithi<br />teuthology-2016-12-21_10:00:22-rbd-jewel-distro-basic-smithi</p>
<p>although, you know, actually, it occurs to me, I only need to check the toplevel directories.</p>
<p>I will scrub the bad ones when I get a full list.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=849112017-01-27T01:04:43ZDan Mickdmick@redhat.com
<ul></ul><p>Duh, ls was fine:</p>
<p>ls -ld * | sort -n -k 5<br />drwxrwxr-x 1 1001 1001 18446744057908416832 Jan 23 02:23 teuthology-2016-12-18_02:01:14-rbd-master-distro-basic-smithi<br />drwxrwxr-x 1 1001 1001 18446744073298154785 Jan 23 04:49 teuthology-2016-12-19_19:25:02-upgrade:hammer-jewel-x-kraken-distro-basic-vps<br />drwxrwxr-x 1 1001 1001 18446744073703811907 Jan 23 00:30 teuthology-2016-12-21_10:00:22-rbd-jewel-distro-basic-smithi</p>
<p>scrubbing those now.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=849122017-01-27T01:07:02ZDan Mickdmick@redhat.com
<ul></ul><pre>
# ceph daemon mds.mira049 scrub_path teuthology-2016-12-18_02:01:14-rbd-master-distro-basic-smithi repair recursive force
{
"return_code": -116
}
</pre> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=849132017-01-27T01:18:13ZDan Mickdmick@redhat.com
<ul></ul><p>They all return ESTALE. Not sure what else I need to be doing</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=850542017-01-28T04:37:18ZDan Mickdmick@redhat.com
<ul></ul><p>Tried them again tonight after repairing the broken stray object, and they worked this time. <shrug></p>
<p>I guess the answer <strong>should</strong> have been scrub_path, but I don't know why sometimes it returns ESTALE.</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=850562017-01-28T11:17:11ZZheng Yanukernel@gmail.com
<ul></ul><p>Dan Mick wrote:</p>
<blockquote>
<p>[...]</p>
</blockquote>
<p>teuthology-2016-12-18_02:01:14-rbd-master-distro-basic-smithi is not in root directory, it's in teuthology-archive</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=851702017-01-30T19:28:18ZDan Mickdmick@redhat.com
<ul></ul><p>Would that have caused ESTALE?</p> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=852322017-01-31T13:01:29ZJohn Sprayjcspray@gmail.com
<ul></ul><p>Current status of lab cluster is:</p>
<ul>
<li>Fixed the "missing dirfrag object" damage with a script that removed the offending omap entries.</li>
<li>scrub_path on teuthology-archive/ to fix the stats, which threw up a load of (imho bogus) "bad backtrace" damage (<a class="external" href="http://tracker.ceph.com/issues/18743">http://tracker.ceph.com/issues/18743</a>)</li>
<li>Restarted the MDS and did not run scrub again, so damage table currently empty</li>
<li>If someone runs scrub again on /teuthology-archive they're liable to see a load of "bad backtrace" damage again, if you see other types of damage then worry</li>
<li>Let's install an updated ceph-mds as soon as <a class="external" href="http://tracker.ceph.com/issues/18743">http://tracker.ceph.com/issues/18743</a> is fixed and backported to kraken.</li>
</ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1106652018-04-09T20:28:44ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Subject</strong> changed from <i>Forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)</i> to <i>mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)</i></li><li><strong>Target version</strong> changed from <i>v12.0.0</i> to <i>v13.0.0</i></li><li><strong>Source</strong> set to <i>Development</i></li><li><strong>Tags</strong> set to <i>scrub</i></li><li><strong>Backport</strong> changed from <i>jewel kraken</i> to <i>jewel,luminous</i></li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1138042018-05-22T22:10:12ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> changed from <i>v13.0.0</i> to <i>v14.0.0</i></li><li><strong>Backport</strong> changed from <i>jewel,luminous</i> to <i>mimic,luminous</i></li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1201302018-09-07T01:52:49ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>4</i> to <i>New</i></li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1309422019-03-07T23:21:08ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> changed from <i>v14.0.0</i> to <i>v15.0.0</i></li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1310612019-03-07T23:32:11ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> deleted (<del><i>v15.0.0</i></del>)</li></ul> CephFS - Bug #18532: mds: forward scrub failing to repair dir stats (was: subdir with corrupted dirstat is un-rm-able)https://tracker.ceph.com/issues/18532?journal_id=1569412020-01-24T23:12:52ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Tags</strong> deleted (<del><i>scrub</i></del>)</li><li><strong>Backport</strong> deleted (<del><i>mimic,luminous</i></del>)</li><li><strong>Labels (FS)</strong> <i>scrub</i> added</li></ul>