Ceph » rbd

The following scenario leads to the split-brain failure:

1) start rbd-mirror on site1 (site1 is local and site2 is remote/peer)
2) create a mirroring image on site2 and observer successful mirroring to site1
3) demote the primary image on site2
4) promote the non-primary image on site1
5) start rbd-mirror on site2 (site2 is local and site1 is remote/peer)

When the started site2 rbd-mirror tries to mirror the image promoted on site1 it enters the split-brain state:

2021-12-02T07:58:32.305+0000 7fc88f4ca640 10 rbd::mirror::image_replayer::journal::PrepareReplayRequest: 0x563ee2642500 handle_get_remote_tags: r=0
2021-12-02T07:58:32.305+0000 7fc88f4ca640 -1 rbd::mirror::image_replayer::journal::PrepareReplayRequest: 0x563ee2642500 handle_get_remote_tags: split-brain detected -- skipping image replay
2021-12-02T07:58:32.305+0000 7fc88f4ca640 10 rbd::mirror::image_replayer::BootstrapRequest: 0x563edb4774a0 handle_prepare_replay: r=-17
2021-12-02T07:58:32.305+0000 7fc88f4ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563edb4774a0 handle_prepare_replay: failed to prepare local replay: (17) File exists

test:
  global_id:   17923ed4-6142-462a-a813-71978ff611c1
  state:       up+error
  description: split-brain detected
  service:     mirror.0 on adonis
  last_update: 2021-12-02 08:10:00
  peer_sites:
    name: cluster1
    state: up+stopped
    description: local image is primary
    last_update: 2021-12-02 08:09:58

Normally, when rbd-mirror is running on both sites, although the image replay in direction from the non-primary image to primary is not running, the rbd-mirror registers itself in the remote (non-primary) image journal. Then after failover, the rbd-mirror properly detects the demotion/promotion state processing the journal tags created during demotion/promotion. But if the rbd-mirror is started for the first time after the demotion/promotion, it registers in the remote image journal starting from the current position and can't see the older tags that it needs to properly detect the state in rbd::mirror::image_replayer::journal::PrepareReplayRequest::handle_get_remote_tags, and enters the split-brain state.