Bug #62596
Updated by Matan Breizman 9 months ago
Clusters affected by the SnapMapper malformed key conversion [1] (which was fixed) may still suffer from space leak caused by stale clone objects. The leak may occur in the following scenario: A cluster which had snapshots taken and was updated from N (and earlier) to O (up to 16.2.11 - *before the fix [2] was merged*). If one of the snapshots which were taken before the update is removed, the clone objects of this snapshot will become stale. Note: Even if non of the snapshots were removed yet, the key is still malformed and any future removal of this snapshot will cause the same effect. (Unless the SnapMapper key is fixed). *** The fix for the affected clusters will include 2-phase procedure: 1) Fixing the key to the correct structure using the newly introduced `fix_malformed_snapmapper_keys` command. Note: An alternative possibility is to re-deploy each osd since the keys can be fixed on recovery as well. structure. 2) Re-removing the already removed snapshots using snapshots. Now that the `force_reremove_snap` command. After the key is fixed, correct, it will be is possible to map the clone objects from the removed snapshot and clean the space leak. This fix is planned to be backported to P final release. Once the fix will be both ready and verified in main - there will be a better estimation on the backport strategy and timing. *** To verify if a cluster was affected, malformed keys can be identified using the `ceph-kvstore-tool`. The following command can be run offline only: <pre><code class="text"> ceph-kvstore-tool bluestore-kv <store-path> list p | grep 'SNA.*_$' </code></pre> *** Disclaimer regarding the scrub "verify SnapMapper consistency" PR [3] which won't be backported to P: The scrub enchantments are only responsible for fixing a corrupted key IFF the snapshot still exists (not removed). If the snapshot was already removed, both the key won't be fixed by scrub and the clone objects will remain stale. *** [1] https://tracker.ceph.com/issues/56147 [2] https://github.com/ceph/ceph/pull/46908 [3] https://github.com/ceph/ceph/pull/47388