The file system is unmounted df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 4096 0 4096 0% /dev tmpfs 98326436 0 98326436 0% /dev/shm tmpfs 39330576 2024 39328552 1% /run /dev/sda1 960302096 496656628 414791044 55% / tmpfs 98326436 20 98326416 1% /tmp /dev/sdb1 1844812308 1319101136 431926136 76% /sdb tmpfs 19665284 4 19665280 1% /run/user/1259 ./bin/rbd --cluster site-a snap ls --all pool1/image1 | tail -n 3 44 .mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.fe85395e-d482-4ac7-9f8b-6d53fffe9491 10 GiB Thu Jun 1 20:19:33 2023 mirror (primary peer_uuids:[c4c2e739-ea54-43b5-958a-b6865aa9f100]) 45 .mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.2f15d11b-7449-4843-9ce7-9536957b2b80 10 GiB Thu Jun 1 20:19:48 2023 mirror (demoted peer_uuids:[c4c2e739-ea54-43b5-958a-b6865aa9f100]) 46 .mirror.non_primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.932571d4-a2eb-4f21-881f-b3535400d574 10 GiB Thu Jun 1 20:20:08 2023 mirror (non-primary peer_uuids:[] 7cc4842a-0342-47ad-aca4-5298f8bbec95:46 copied) ./bin/rbd --cluster site-b snap ls --all pool1/image1 --debug-rbd 0 44 .mirror.non_primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.96534a3a-25a6-4753-ba09-28f980ac6108 10 GiB Thu Jun 1 20:19:34 2023 mirror (non-primary peer_uuids:[] e992fded-15fd-41f1-89fe-8205afb5a250:44 copied) 45 .mirror.non_primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.2f10427b-a7d6-45ac-b02e-d2ea360baee1 10 GiB Thu Jun 1 20:19:49 2023 mirror (demoted peer_uuids:[db864132-f6f8-430e-888e-a0a72677d51a] e992fded-15fd-41f1-89fe-8205afb5a250:45 copied) 46 .mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.8fb5ab26-d9c1-4d3f-adf4-3ab16fcf3be0 10 GiB Thu Jun 1 20:20:04 2023 mirror (primary peer_uuids:[db864132-f6f8-430e-888e-a0a72677d51a]) sudo ./bin/rbd --cluster site-a device map pool1/image1@.mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.2f15d11b-7449-4843-9ce7-9536957b2b80 /dev/rbd4 fsck -fn /dev/rbd4 fsck from util-linux 2.38.1 e2fsck 1.46.5 (30-Dec-2021) fsck.ext2: Permission denied while trying to open /dev/rbd4 You must have r/o access to the filesystem or be root [choffman@vossi05 build]$ sudo fsck -fn /dev/rbd4 fsck from util-linux 2.38.1 e2fsck 1.46.5 (30-Dec-2021) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/rbd4: 71133/655360 files (0.0% non-contiguous), 343122/2621440 blocks sudo ./bin/rbd --cluster site-a device unmap pool1/image1@.mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.2f15d11b-7449-4843-9ce7-9536957b2b80 sudo ./bin/rbd --cluster site-b device map pool1/image1@.mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.8fb5ab26-d9c1-4d3f-adf4-3ab16fcf3be0 /dev/rbd4 sudo fsck -fn /dev/rbd4 fsck from util-linux 2.38.1 e2fsck 1.46.5 (30-Dec-2021) Pass 1: Checking inodes, blocks, and sizes HTREE directory inode 262217 has an invalid root node. Clear HTree index? no HTREE directory inode 263239 has an invalid root node. Clear HTree index? no Pass 2: Checking directory structure Directory inode 262145, block #0, offset 0: directory corrupted Salvage? no e2fsck: aborted /dev/rbd4: ********** WARNING: Filesystem still has errors ********** [choffman@vossi05 build]$ sudo ./bin/rbd --cluster site-b device unmap pool1/image1@.mirror.primary.5b41f828-1f40-40cd-be09-ce45ed46ee20.8fb5ab26-d9c1-4d3f-adf4-3ab16fcf3be0