Ceph » rbd

This bug creates holes (zeros in this case) in the image where there should be data. This causes corruption on the block and filesystem level. A reproducer was created to illustrate the steps to cause the problem and show associated symptoms.

The environment:

There's two sites, site-a and site-b. To start the scenario, an image is created with mirror mode.

Here's how we arrive at the bug:

1. Maps image on Primary
2. Formats then mount bdev
3a. starts manually creating 40 mirror image snapshots, every 3 seconds. This step will end 3b when completed.  
3b. untars via script provided with timeout of 5m
4. unmap then demote image on primary
5. wait for demote image to be synced to secondary
6. promote image on secondary
7. map image on secondary
8. fsck -fn image

After running the steps above a data corruption occurred.
This can be seen running fsck on the volume:

+ sudo fsck -fn /dev/rbd0
fsck from util-linux 2.38.1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
HTREE directory inode 83315 has an invalid root node.
Clear HTree index? no

HTREE directory inode 83315 has an unsupported hash version (20)
Clear HTree index? no

HTREE directory inode 83315 uses an incompatible htree root node flag.
Clear HTree index? no
...

Steps to reproduce are:
1. In the build directory of a clone of upstream run:

sh ./rbd_reproducer_corrupt1.sh

2. This script will run until error code is returned (most likely going to be fsck).