https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2021-10-18T20:01:14ZCeph rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2044932021-10-18T20:01:14ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li><li><strong>Target version</strong> deleted (<del><i>v16.2.6</i></del>)</li><li><strong>Tags</strong> set to <i>gc</i></li><li><strong>Affected Versions</strong> <i>v16.2.6</i> added</li></ul><p>how big was this object?</p>
<p>are you familiar with the garbage collection system that cleans up these deleted tail objects? see <a class="external" href="https://docs.ceph.com/en/latest/radosgw/config-ref/#garbage-collection-settings">https://docs.ceph.com/en/latest/radosgw/config-ref/#garbage-collection-settings</a></p>
<p>you can use `radosgw-admin gc list --include-all` to show all files that are scheduled for garbage collection. by default, garbage collection waits 2 hours (rgw_gc_obj_min_wait) before it cleans up these objects, because clients may have started reading them just before deletion</p>
<p>note that there is a gc bug when objects get too large (like over 1TB) that we're tracking at <a class="external" href="https://tracker.ceph.com/issues/49823">https://tracker.ceph.com/issues/49823</a></p> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2045052021-10-19T09:11:07ZSemyon Poklad
<ul></ul><p>Casey Bodley wrote:</p>
<blockquote>
<p>how big was this object?</p>
<p>are you familiar with the garbage collection system that cleans up these deleted tail objects? see <a class="external" href="https://docs.ceph.com/en/latest/radosgw/config-ref/#garbage-collection-settings">https://docs.ceph.com/en/latest/radosgw/config-ref/#garbage-collection-settings</a></p>
<p>you can use `radosgw-admin gc list --include-all` to show all files that are scheduled for garbage collection. by default, garbage collection waits 2 hours (rgw_gc_obj_min_wait) before it cleans up these objects, because clients may have started reading them just before deletion</p>
<p>note that there is a gc bug when objects get too large (like over 1TB) that we're tracking at <a class="external" href="https://tracker.ceph.com/issues/49823">https://tracker.ceph.com/issues/49823</a></p>
</blockquote>
<p>Thanks, @Casey Bodley! File ~ 300mb ."radosgw-admin --bucket = support-files gc process --include-all" actually cleared the pool. So this is not a bug.</p>
<p>Found another problem. I was creating a test pool and bucket that I already deleted along with the objects - bucket and pool created from dasboard, objects deleted from rados -p rm. Bucket deleted from dashboard, pool from gui pve</p>
<p>But<br />"radosgw-admin gc list --include-all" shows this<br /><pre><code class="c syntaxhl"><span class="CodeRay"> {
<span class="string"><span class="delimiter">"</span><span class="content">pool</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">test.bucket.data</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">oid</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2~nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">key</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">instance</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>
}
</span></code></pre></p>
<p>radosgw-admin process --include-all --debug-ms = 1 does not output errors, but rerun gc list does not show changes<br />The collector sees non-existent objects. radosgw-admin bucket check --fix did not affect the result<br />Since the objects in the default pool were manually deleted many times, I created the test.bucket.data pool again and then ran the gc process -> gc list -clean! <br />It is not obvious behavior that the collector tries to work with non-existent objects and pools and does not give an error message.</p> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2047082021-10-21T14:25:55ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Subject</strong> changed from <i>pacific_rgw: when deleting large files, objects *multipart* and *shadows* are not deleted but become orphans </i> to <i>garbage collection doesn't remove gc list entries if the object's pool doesn't exist</i></li></ul> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2047092021-10-21T14:26:29ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Pritha Srivastava</i></li><li><strong>Backport</strong> set to <i>octopus pacific</i></li></ul> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2047102021-10-21T14:27:09ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>New</i></li></ul> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2047112021-10-21T14:27:17ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Triaged</i></li></ul> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2049042021-10-27T04:59:43ZPritha Srivastavaprsrivas@redhat.com
<ul><li><strong>Status</strong> changed from <i>Triaged</i> to <i>Need More Info</i></li></ul><p>Can you give me the exact steps that you tried for the second bug that you are reporting,</p>
<p>I understand that you have a test pool, and then you created and objects. And then you deleted them, and when you run a gc list, it gives some o/p, but when you run gc process, it doesn't delete anything? and you still see the same o/p from gc list command? And I understand that these are multipart objects. Is there anything else that you would like to add to this?</p> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2049602021-10-27T14:27:46ZSemyon Poklad
<ul></ul><p>Pritha Srivastava wrote:</p>
<blockquote>
<p>Can you give me the exact steps that you tried for the second bug that you are reporting,</p>
<p>I understand that you have a test pool, and then you created and objects. And then you deleted them, and when you run a gc list, it gives some o/p, but when you run gc process, it doesn't delete anything? and you still see the same o/p from gc list command? And I understand that these are multipart objects. Is there anything else that you would like to add to this?</p>
</blockquote>
<p>1.Created pool test.bucket.data<br />2.Created a bucket in this pool<br />3.Added objects to this bucket<br />4.Deleted objects (First through the browser and then rados -p rm)<br />5.Removed the bucket<br />6.Removed pool test.bucket.data</p>
<p>7. "radosgw-admin gc list --include-all" shows this</p>
<pre><code>{<br /> "pool": "test.bucket.data",<br /> "oid": "0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2 ~ nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48",<br /> "key": "",<br /> "instance": "" <br /> }<br />8. "radosgw-admin gc process --include-all" and "radosgw-admin bucket check --fix" -No error messages<br />9. Again I do "radosgw-admin gc list --include-all" and again I see this
{<br /> "pool": "test.bucket.data",<br /> "oid": "0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2 ~ nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48",<br /> "key": "",<br /> "instance": "" <br /> }<br />10.radosgw-admin process --include-all --debug-ms = 1 does not output errors<br />11.radosgw-admin gc list --include-all everything is still<br />12.Create pool test.bucket.data<br />13.radosgw-admin process --include-all<br />14.radosgw-admin gc list --include-all output is now clean.</code></pre> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2050202021-10-28T14:16:04ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>New</i></li></ul> rgw - Bug #52964: garbage collection doesn't remove gc list entries if the object's pool doesn't existhttps://tracker.ceph.com/issues/52964?journal_id=2050212021-10-28T14:16:15ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Triaged</i></li></ul>