Bug #58852
Updated by Ilya Dryomov about 1 year ago
<pre> $ rbd --cluster site-a create -s 100M --size 100 data/img $ rbd --cluster site-a mirror image enable data/img snapshot Mirroring enabled $ sudo rbd map --cluster site-a -t nbd data/img /dev/nbd0 $ xfs_io -d -c 'pwrite -b 4M 0 100M' /dev/nbd0 wrote 104857600/104857600 bytes at offset 0 100 MiB, 25 ops; 0:00:01.42 (69.952 MiB/sec and 17.4879 ops/sec) $ rbd --cluster site-a du data/img NAME PROVISIONED USED img 100 MiB 100 MiB $ rbd --cluster site-a mirror image snapshot data/img Snapshot ID: 20 < ... snapshot sync ... > $ rbd --cluster site-b du data/img NAME PROVISIONED USED img 100 MiB 100 MiB $ blkdiscard -l 80M /dev/nbd0 $ rbd --cluster site-a du data/img NAME PROVISIONED USED img 100 MiB 20 MiB $ rbd --cluster site-a mirror image snapshot data/img Snapshot ID: 22 < ... snapshot sync ... > $ rbd --cluster site-b du data/img NAME PROVISIONED USED img 100 MiB 100 MiB </pre> It's not just that the area remains allocated on site-b, it still contains data at those offsets. The checksums don't match, of course: <pre> $ xfs_io -d -c 'pread -v 0 512' /dev/nbd0 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ read 512/512 bytes at offset 0 512.000000 bytes, 1 ops; 0.0016 sec (309.789 KiB/sec and 619.5787 ops/sec) $ md5sum /dev/nbd0 067122afb788aa6349f7780dadbf9811 /dev/nbd0 $ sudo rbd unmap --cluster site-a -t nbd data/img $ rbd --cluster site-a mirror image demote data/img Image demoted to non-primary $ rbd --cluster site-b mirror image promote data/img Image promoted to primary $ sudo rbd map --cluster site-b -t nbd data/img /dev/nbd0 $ xfs_io -d -c 'pread -v 0 512' /dev/nbd0 00000000: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000010: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000020: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000030: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000040: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000050: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000060: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000070: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000080: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000090: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000a0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000b0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000c0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000d0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000e0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000000f0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000100: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000110: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000120: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000130: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000140: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000150: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000160: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000170: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000180: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000190: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001a0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001b0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001c0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001d0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001e0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001f0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ read 512/512 bytes at offset 0 512.000000 bytes, 1 ops; 0.0014 sec (353.357 KiB/sec and 706.7138 ops/sec) $ md5sum /dev/nbd0 a5547703c21e4855c5e3e4abc5a668d6 /dev/nbd0 </pre>