Actions
Bug #61616
closed[librbd] volume data corruption when using rbd-mirror w/failover
% Done:
0%
Source:
Tags:
backport_processed
Backport:
pacific,quincy,reef
Regression:
Yes
Severity:
3 - minor
Reviewed:
Description
This bug creates holes (zeros in this case) in the image where there should be data. This causes corruption on the block and filesystem level. A reproducer was created to illustrate the steps to cause the problem and show associated symptoms.
The environment:
There's two sites, site-a and site-b. To start the scenario, an image is created with mirror mode.
Here's how we arrive at the bug:
1. Maps image on Primary 2. Formats then mount bdev 3a. starts manually creating 40 mirror image snapshots, every 3 seconds. This step will end 3b when completed. 3b. untars via script provided with timeout of 5m 4. unmap then demote image on primary 5. wait for demote image to be synced to secondary 6. promote image on secondary 7. map image on secondary 8. fsck -fn image
After running the steps above a data corruption occurred.
This can be seen running fsck on the volume:
+ sudo fsck -fn /dev/rbd0 fsck from util-linux 2.38.1 e2fsck 1.46.5 (30-Dec-2021) Pass 1: Checking inodes, blocks, and sizes HTREE directory inode 83315 has an invalid root node. Clear HTree index? no HTREE directory inode 83315 has an unsupported hash version (20) Clear HTree index? no HTREE directory inode 83315 uses an incompatible htree root node flag. Clear HTree index? no ...
Steps to reproduce are:
1. In the build directory of a clone of upstream run:
sh ./rbd_reproducer_corrupt1.sh
2. This script will run until error code is returned (most likely going to be fsck).
Files
Actions