snapshots will not be deleted after upgrade from nautilus to pacific
After upgrading from 14.2.22 to 16.2.9 snapshot deletion does not remove "clones" from pool
More precise: Objects in snapshots created with nautilus and deleted with pacific. New created snapshots working as expected.
The snapshot itself disappears in the snapshot list but the number of "clones" listed in rados df will not be cleared and the corresponding space will not be freed up.
Also I could this observer on a update from nautilus to ocotopus.
It doesn't matter if the omap conversion was done with the update or not (bluestore_fsck_quick_fix_on_mount)
#2 Updated by Manuel Lausch 8 months ago
Yes. For the debuglogs I tested this with nautilus (14.2.22) to octopus (15.2.16). The behavior is the same as described before with update von Nautilus to Pacific
debug logs are uploaded
The testcluster is a singlenode with 3 OSDs and a replicated pool with rf=3
on nautilus I created 4 objects
created a snapshot
delete one object
here the output from rados df so far:
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR spielfeld 768 KiB 5 1 15 0 0 0 1 1 KiB 7 5 KiB 0 B 0 B total_objects 5 total_used 3.0 GiB total_avail 11 TiB total_space 11 TiB
now I updated to octopus and deleted the snapshot again
The rados df output is still the same:
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR device_health_metrics 57 KiB 3 0 9 0 0 0 0 0 B 3 15 KiB 0 B 0 B spielfeld 768 KiB 5 1 15 0 0 0 0 0 B 7 5 KiB 0 B 0 B total_objects 8 total_used 3.0 GiB total_avail 11 TiB total_space 11 TiB
#3 Updated by Manuel Lausch 7 months ago
It seems to be a failure on conversion after upgrade
in the omap dump before the update with one deleted object in a snapshot there are this two entries
_USER_0000000000000065_USER_ MAP_0000000000000001_0000000000000001.F13BC082.1.d1.. _USER_0000000000000065_USER_ OBJ_0000000000000001.F13BC082.1.d1..
After the update to octopus, the entries looks like this:
_USER_0000000000000065_USER_ OBJ_0000000000000001.F13BC082.1.d1.. _USER_0000000000000065_USER_ SNA_1_0000000000000001_
With a octopus deleted object, the entry looks like this:
In the debug output of the OSD there is only one log line, which says, there was one key converted. No more details.
#6 Updated by Matan Breizman 7 months ago
Radoslaw Zarzynski wrote:
Hello Matan! Does this snapshot issue ring a bell?
Here I have a PR, which should fix the conversion on update
But what is with clusters already updated?
This patch may work when running Octopus for the first time.
As mentioned, we should also be able to convert already updated clusters.
#13 Updated by Matan Breizman 7 months ago
This issue is fixed (including a unit test) and will be backported in order to prevent future clusters upgrades from Nautilus (or earlier) to use the faulty conversion.
For already-converted clusters: Separate PR will be issued to remove/update the malformed SnapMapper keys.