Ceph » RADOS

https://tracker.ceph.com/issues/62983 should help with avoiding the gaps in the purged snaps ids intervals. As a result, all the purged_snap ids will be mereged into a single entry.

For clusters already impacted by this issue, https://github.com/ceph/ceph/pull/53545 may help with removing the "ghost" snapids which cause the gap and allow merging all the entries.

Adding 53545 as a candidate for fixing this issue, this will require additional documentation on how to use the tool - so I'll keep the tracker open.

Eugen Block wrote in #note-6:

I know I'm a bit early asking this, but I helped raise this issue and Mykola picked it up in the devel mailing list. I talked to one of our customers who is affected by this (more than 40 Million purged_snap entries) and they would be interested testing this feature on their secondary site (they mirror rbd images). But they're currently in the planning process to remove the second site, I have no ETA though. I expect this fix (and the respective tool to trim affected mon stores) not earlier than for Squid. There's no telling if and when they will be able to upgrade to Squid, they recently upgraded to Quincy though. So would there be a chance to backport this to Reef and Quincy as well, depending on which release they'll be on when this is considered ready?

Hey Eugen,
There should be no issues with backporting this back to Q as this PR offers a new separated command.
The relevant usage of the command will be using with the default option:

     *  * Default: All the snapids in the given range which are not
     *    marked as purged in the Monitor will be removed. Mostly useful
     *    for cases in which the snapid is leaked in the client side.
     *    See: https://tracker.ceph.com/issues/64646

And a couple more questions regarding the purge tool:

1. Will it be possible to trim the keys online (without cluster downtime)?
2. How "safe" will it be? What could go wrong and would there be some rollback mechanism?

1. Yes, it's possible. The (online) command doesn't require shutting down the OSDs or MONs.
2. The command was also added to our testing workloads and seem to work well.
   The tricky part is the unknown unknowns. I do not expect anything to go wrong as the command will only interact with ghost snapids. Moreover, the command can also be used gradually (short snapid removal intervals) to verify nothing goes wrong while using it.