Ceph » rbd

A mirrored image lost an in-use snapshot somehow before it was done using it. This resulted in an assertion failure when attempting to load the object-map for the non-existent snapshot that had been deleted. The snapshot should not have been able to be deleted, though, since it should have still be tied to the peer:

#8  0x0000563574a7bff8 in librbd::object_map::DiffRequest<librbd::ImageCtx>::load_object_map (this=this@entry=0x5635a1fad500, image_locker=image_locker@entry=0x7f6a70e9be00) at /usr/include/c++/10/bits/stl_tree.h:1010
#9  0x0000563574a7cc10 in librbd::object_map::DiffRequest<librbd::ImageCtx>::send (this=0x5635a1fad500) at /usr/include/c++/10/bits/stl_vector.h:1792
#10 0x00005635747d0977 in rbd::mirror::image_replayer::snapshot::Replayer<librbd::ImageCtx>::handle_request_sync (this=0x56358eb28c00, r=0) at /usr/include/c++/10/x86_64-redhat-linux/bits/gthr-default.h:779

State machine log:

2020-12-09T19:03:55.362-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 Replayer: 
2020-12-09T19:03:55.362-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 init: 
2020-12-09T19:03:55.362-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 init: remote_mirror_peer_uuid=61a576e8-a1cc-46d7-befc-3b7c82ebc12f
2020-12-09T19:03:55.362-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 register_local_update_watcher: 
2020-12-09T19:03:55.373-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_register_local_update_watcher: r=0
2020-12-09T19:03:55.373-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 register_remote_update_watcher: 
2020-12-09T19:03:55.389-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_register_remote_update_watcher: r=0
2020-12-09T19:03:55.389-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_status_updated: 
2020-12-09T19:03:55.389-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 load_local_image_meta: 
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_load_local_image_meta: r=-2
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: 
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: 
2020-12-09T19:03:55.396-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=28, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:55.396-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=2028, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: found remote mirror snapshot: remote_snap_id_start=0, remote_snap_id_end=28, remote_snap_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_snapshots: remote_snap_id_start=0, remote_snap_id_end=28, local_snap_id_start=0
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: r=0
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: remote_snap_id_start=0, remote_snap_id_end=28, local_snap_id_start=0, snap_seqs={28=18446744073709551614}
2020-12-09T19:03:55.396-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 get_remote_image_state: 
2020-12-09T19:03:55.397-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_get_remote_image_state: r=0
2020-12-09T19:03:55.397-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 create_non_primary_snapshot: demoted=0, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=28, snap_seqs={28=18446744073709551614}
2020-12-09T19:03:56.267-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: r=0
2020-12-09T19:03:56.267-0500 7f6a6a693700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: local_snap_id_end=1710
2020-12-09T19:03:56.267-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 request_sync: 
2020-12-09T19:03:58.862-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_request_sync: r=0
2020-12-09T19:03:58.862-0500 7f6a7069f700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_image: remote_snap_id_start=0, remote_snap_id_end=28, local_snap_id_start=0, last_copied_object_number=0, snap_seqs={28=18446744073709551614}
2020-12-09T19:03:58.862-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_image: r=0
2020-12-09T19:03:58.862-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 apply_image_state: 
2020-12-09T19:03:58.863-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_apply_image_state: r=0
2020-12-09T19:03:58.863-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 update_non_primary_snapshot: 
2020-12-09T19:03:58.863-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_update_non_primary_snapshot: r=0
2020-12-09T19:03:58.863-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_image_update: 
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_notify_image_update: r=0
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 finish_sync: 
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_status_updated: 
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 load_local_image_meta: 
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_load_local_image_meta: r=-2
2020-12-09T19:03:58.863-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 refresh_local_image: 
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_refresh_local_image: r=0
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: 
2020-12-09T19:03:58.864-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: local mirror snapshot: id=1710, mirror_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=1c, last_copied_object_number=0, snap_seqs={28=18446744073709551614}]
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: found local mirror snapshot: local_snap_id_start=1710, local_snap_id_end=18446744073709551614, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=1c, last_copied_object_number=0, snap_seqs={28=18446744073709551614}]
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: 
2020-12-09T19:03:58.864-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=28, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:58.864-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: skipping synced remote snapshot 28
2020-12-09T19:03:58.864-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=2028, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: found remote mirror snapshot: remote_snap_id_start=28, remote_snap_id_end=2028, remote_snap_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_snapshots: remote_snap_id_start=28, remote_snap_id_end=2028, local_snap_id_start=1710
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: r=0
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: remote_snap_id_start=28, remote_snap_id_end=2028, local_snap_id_start=1710, snap_seqs={2028=18446744073709551614}
2020-12-09T19:03:58.864-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 get_remote_image_state: 
2020-12-09T19:03:58.864-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_get_remote_image_state: r=0
2020-12-09T19:03:58.864-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 create_non_primary_snapshot: demoted=0, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=2028, snap_seqs={2028=18446744073709551614}
2020-12-09T19:03:59.158-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: r=0
2020-12-09T19:03:59.158-0500 7f6a6a693700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: local_snap_id_end=3919
2020-12-09T19:03:59.158-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 request_sync: 
2020-12-09T19:10:03.879-0500 7f6a70ea0700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_request_sync: r=0
2020-12-09T19:10:03.879-0500 7f6a70ea0700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_image: remote_snap_id_start=28, remote_snap_id_end=2028, local_snap_id_start=1710, last_copied_object_number=0, snap_seqs={2028=18446744073709551614}
2020-12-09T19:10:03.880-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_image_progress: object_number=1, object_count=1
2020-12-09T19:10:03.880-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 update_non_primary_snapshot: 
2020-12-09T19:10:03.880-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_image: r=0
2020-12-09T19:10:03.880-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 apply_image_state: 
2020-12-09T19:10:03.881-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_update_non_primary_snapshot: r=0
2020-12-09T19:10:03.882-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_apply_image_state: r=0
2020-12-09T19:10:03.882-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 update_non_primary_snapshot: 
2020-12-09T19:10:03.883-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_update_non_primary_snapshot: r=0
2020-12-09T19:10:03.883-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_image_update: 
2020-12-09T19:10:03.883-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_notify_image_update: r=0
2020-12-09T19:10:03.883-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 unlink_peer: remote_snap_id=28
2020-12-09T19:10:04.876-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_unlink_peer: r=0
2020-12-09T19:10:04.876-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 finish_sync: 
2020-12-09T19:10:04.876-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_status_updated: 
2020-12-09T19:10:04.876-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 load_local_image_meta: 
2020-12-09T19:10:04.877-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_load_local_image_meta: r=-2
2020-12-09T19:10:04.877-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 refresh_local_image: 
2020-12-09T19:10:04.879-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_refresh_local_image: r=0
2020-12-09T19:10:04.879-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 refresh_remote_image: 
2020-12-09T19:10:04.879-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_refresh_remote_image: r=0
2020-12-09T19:10:04.879-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: 
2020-12-09T19:10:04.879-0500 7f6a63e86700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: local mirror snapshot: id=1710, mirror_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=1c, last_copied_object_number=0, snap_seqs={28=18446744073709551614}]
2020-12-09T19:10:04.879-0500 7f6a63e86700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: local mirror snapshot: id=3919, mirror_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=7ec, last_copied_object_number=1, snap_seqs={2028=18446744073709551614}]
2020-12-09T19:10:04.879-0500 7f6a63e86700  5 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: pruning unused non-primary snapshot 1710
2020-12-09T19:10:04.879-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 prune_non_primary_snapshot: snap_id=1710
2020-12-09T19:10:05.079-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_prune_non_primary_snapshot: r=0
2020-12-09T19:10:05.079-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 load_local_image_meta: 
2020-12-09T19:10:05.080-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_load_local_image_meta: r=-2
2020-12-09T19:10:05.080-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 refresh_local_image: 
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_refresh_local_image: r=0
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: 
2020-12-09T19:10:05.081-0500 7f6a6a693700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: local mirror snapshot: id=3919, mirror_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=7ec, last_copied_object_number=1, snap_seqs={2028=18446744073709551614}]
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: found local mirror snapshot: local_snap_id_start=3919, local_snap_id_end=18446744073709551614, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=7ec, last_copied_object_number=1, snap_seqs={2028=18446744073709551614}]
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: 
2020-12-09T19:10:05.081-0500 7f6a6a693700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=4370, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:10:05.081-0500 7f6a6a693700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=7873, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: unlinking from remote snapshot 7873
2020-12-09T19:10:05.081-0500 7f6a6a693700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 unlink_peer: remote_snap_id=7873
2020-12-09T19:10:05.083-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_unlink_peer: r=0
2020-12-09T19:10:05.083-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 finish_sync: 
2020-12-09T19:10:05.083-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 notify_status_updated: 
2020-12-09T19:10:05.083-0500 7f6a63e86700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 load_local_image_meta: 
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_load_local_image_meta: r=-2
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: 
2020-12-09T19:10:05.084-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: local mirror snapshot: id=3919, mirror_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=7ec, last_copied_object_number=1, snap_seqs={2028=18446744073709551614}]
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_local_mirror_snapshots: found local mirror snapshot: local_snap_id_start=3919, local_snap_id_end=18446744073709551614, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=7ec, last_copied_object_number=1, snap_seqs={2028=18446744073709551614}]
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: 
2020-12-09T19:10:05.084-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=4370, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:10:05.084-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: remote mirror snapshot: id=7873, mirror_ns=[mirror state=primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:10:05.084-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: skipping remote snapshot due to missing mirror peer
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 scan_remote_mirror_snapshots: found remote mirror snapshot: remote_snap_id_start=2028, remote_snap_id_end=4370, remote_snap_ns=[mirror state=primary, complete=1, mirror_peer_uuids=61a576e8-a1cc-46d7-befc-3b7c82ebc12f, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_snapshots: remote_snap_id_start=2028, remote_snap_id_end=4370, local_snap_id_start=3919
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: r=0
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_copy_snapshots: remote_snap_id_start=2028, remote_snap_id_end=4370, local_snap_id_start=3919, snap_seqs={4370=18446744073709551614}
2020-12-09T19:10:05.084-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 get_remote_image_state: 
2020-12-09T19:10:05.084-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_get_remote_image_state: r=0
2020-12-09T19:10:05.084-0500 7f6a63685700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 create_non_primary_snapshot: demoted=0, primary_mirror_uuid=a2522fda-480a-4337-aefa-ab17c73941b5, primary_snap_id=4370, snap_seqs={4370=18446744073709551614}
2020-12-09T19:10:06.082-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: r=0
2020-12-09T19:10:06.082-0500 7f6a69e92700 15 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_create_non_primary_snapshot: local_snap_id_end=7707
2020-12-09T19:10:06.082-0500 7f6a69e92700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 request_sync: 
2020-12-09T19:16:43.964-0500 7f6a70ea0700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 handle_request_sync: r=0
2020-12-09T19:16:43.964-0500 7f6a70ea0700 10 rbd::mirror::image_replayer::snapshot::Replayer: 0x56358eb28c00 copy_image: remote_snap_id_start=2028, remote_snap_id_end=4370, local_snap_id_start=3919, last_copied_object_number=0, snap_seqs={4370=18446744073709551614}