Ceph » RADOS

Yes. For the debuglogs I tested this with nautilus (14.2.22) to octopus (15.2.16). The behavior is the same as described before with update von Nautilus to Pacific

debug logs are uploaded
ceph-post-file: 9a210755-bcc5-4b8b-bca5-45bf0071b142

The testcluster is a singlenode with 3 OSDs and a replicated pool with rf=3

on nautilus I created 4 objects
created a snapshot
delete one object

here the output from rados df so far:

POOL_NAME     USED  OBJECTS  CLONES  COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS     RD  WR_OPS     WR  USED COMPR  UNDER COMPR
spielfeld  768 KiB        5       1      15                   0        0         0       1  1 KiB       7  5 KiB         0 B          0 B

total_objects    5
total_used       3.0 GiB
total_avail      11 TiB
total_space      11 TiB

now I updated to octopus and deleted the snapshot again
The rados df output is still the same:

POOL_NAME                 USED  OBJECTS  CLONES  COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS   RD  WR_OPS      WR  USED COMPR  UNDER COMPR
device_health_metrics   57 KiB        3       0       9                   0        0         0       0  0 B       3  15 KiB         0 B          0 B
spielfeld              768 KiB        5       1      15                   0        0         0       0  0 B       7   5 KiB         0 B          0 B

total_objects    8
total_used       3.0 GiB
total_avail      11 TiB
total_space      11 TiB

It seems to be a failure on conversion after upgrade

in the omap dump before the update with one deleted object in a snapshot there are this two entries

_USER_0000000000000065_USER_    MAP_0000000000000001_0000000000000001.F13BC082.1.d1..
_USER_0000000000000065_USER_    OBJ_0000000000000001.F13BC082.1.d1..

After the update to octopus, the entries looks like this:

_USER_0000000000000065_USER_    OBJ_0000000000000001.F13BC082.1.d1..
_USER_0000000000000065_USER_    SNA_1_0000000000000001_

With a octopus deleted object, the entry looks like this:

_USER_0000000000000065_USER_    SNA_1_0000000000000001_0000000000000001.A5224EFB.1.d2..

In the debug output of the OSD there is only one log line, which says, there was one key converted. No more details.

Here I have a PR, which should fix the conversion on update
https://github.com/ceph/ceph/pull/46908

But what is with clusters already updated?

Radoslaw Zarzynski wrote:

Hello Matan! Does this snapshot issue ring a bell?

Introduced here:
https://github.com/ceph/ceph/commit/94ebe0eab968068c29fdffa1bfe68c72122db633

Here I have a PR, which should fix the conversion on update
https://github.com/ceph/ceph/pull/46908

But what is with clusters already updated?

This patch may work when running Octopus for the first time.
As mentioned, we should also be able to convert already updated clusters.

This issue is fixed (including a unit test) and will be backported in order to prevent future clusters upgrades from Nautilus (or earlier) to use the faulty conversion.

For already-converted clusters: Separate PR will be issued to remove/update the malformed SnapMapper keys.

Is this bug also affecting rbd snapshots / clones?

Stefan Kooman wrote:

Is this bug also affecting rbd snapshots / clones?

Yes

For already-converted clusters: Separate PR will be issued to remove/update the malformed SnapMapper keys.

https://github.com/ceph/ceph/pull/47388

Matan Breizman wrote:

For already-converted clusters: Separate PR will be issued to remove/update the malformed SnapMapper keys.

https://github.com/ceph/ceph/pull/47388

This PR never got backported to any release. We suspect we may be suffering from corrupted snapshots due this bug. I've create backport request for the above pr. Parallel to this we are trying to gather the information to either prove or disprove the corruption relationship between the bug and our corruption case.

https://tracker.ceph.com/issues/59478