Ceph » rbd

The ceph-immutable-object-cache does not have a special handling for non existent objects, storing them as empty objects and returning an empty result.

When used with rbd it causes issue for layered (clone of clone) images, a read for which may return an empty result instead of trying the parent.

Steps to reproduce:

1) Prepare double clone:

  % rbd create test1 -s 1M
  % rbd bench --io-type write --io-size 1M --io-total 1M test1
  % rbd snap create test1@snap
  % rbd snap protect test1@snap
  % rbd clone test1@snap test2
  % rbd snap create test2@snap
  % rbd snap protect test2@snap
  % rbd clone test2@snap test3

2) Configure immutable_object_cache_path and
   immutable_object_cache_sock in ceph.conf and start the cache controller:

  % ceph-immutable-object-cache --id admin

3) Read the image:

  % rbd export test3 - | xxd| tail -1
  Exporting image: 100% complete...done.
  000ffff0: 6767 6767 6767 6767 6767 6767 6767 6767  gggggggggggggggg
  % rbd export --rbd_parent_cache_enabled=true --rbd_plugins=parent_cache test3 - | xxd| tail -1
  Exporting image: 100% complete...done.
  000ffff0: 6767 6767 6767 6767 6767 6767 6767 6767  gggggggggggggggg
  % rbd export --rbd_parent_cache_enabled=true --rbd_plugins=parent_cache test3 - | xxd| tail -1
  Exporting image: 100% complete...done.
  000ffff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Note, the first read, when using the parent cache returns the correct result, because there is no object in the cache yet. This reads triggers the object promotion (empty file is created) and the next reads return incorrect empty result.

The issue was incidentally detected when running TestDeepCopy.Clone with parent_cache enabled: when comparing the source and the destination images the read from the source image was correct but triggered the empty object promotion resulted in incorrect read from the destination image.