Project

General

Profile

Actions
Copy link

Bug #61707

closed

[rbd-mirror] demote snapshot does not get removed

Added by Christopher Hoffman 11 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Ilya Dryomov
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
53251
Crash signature (v1):
Crash signature (v2):

Description

During a failover of a rbd image,the demoted snapshot does not get cleaned up/removed. After many failovers back and forth there can many demoted snapshots that do not need to be there anymore.

Steps to reproduce:
-Environment needs two sites, named site-a and site-b
-Run the attached script.

After running the attached reproducer script, one will see output like this:

./bin/rbd --cluster site-a snap ls --all pool1/image1 
SNAPID  NAME                                                                                           SIZE      PROTECTED  TIMESTAMP                 NAMESPACE                                                                     
     5  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.9e3f3ad6-d03c-4bfb-aa3a-9ea8e67de6f8      1000 MiB             Thu Jun 15 22:30:01 2023  mirror (demoted peer_uuids:[])                                                
     7  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.037d605b-0ad0-439c-a22e-909806d12302      1000 MiB             Thu Jun 15 22:30:28 2023  mirror (demoted peer_uuids:[])                                                
    14  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.a588283c-4b53-416e-89dd-f6e828d1b0a3      1000 MiB             Thu Jun 15 22:30:58 2023  mirror (demoted peer_uuids:[])                                                
    21  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.5c4fdf8c-65b1-49a4-a2e6-2a44c0af8e45      1000 MiB             Thu Jun 15 22:31:28 2023  mirror (demoted peer_uuids:[])                                                
    28  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.10a042b1-49c0-46eb-bbbc-19fc0401663d      1000 MiB             Thu Jun 15 22:31:58 2023  mirror (demoted peer_uuids:[])                                                
    35  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.5be1e477-ceab-4a2e-82c7-e7cefd6d5e7e      1000 MiB             Thu Jun 15 22:32:29 2023  mirror (demoted peer_uuids:[])                                                
    42  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.c58330e3-34b0-4e49-a76e-571515c6a218      1000 MiB             Thu Jun 15 22:32:58 2023  mirror (demoted peer_uuids:[])                                                
    49  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.32164e8b-aec3-4d35-b78f-d4e01958644a      1000 MiB             Thu Jun 15 22:33:29 2023  mirror (demoted peer_uuids:[])                                                
    56  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.30816f2d-16ce-4a73-976c-b2c12f8a3c55      1000 MiB             Thu Jun 15 22:33:59 2023  mirror (demoted peer_uuids:[])                                                
    63  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.2e3dd5b5-5df9-4fef-b30d-4ecb32e59ecb      1000 MiB             Thu Jun 15 22:34:29 2023  mirror (demoted peer_uuids:[])                                                
    70  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.d856528d-0330-4368-adeb-5c6252d1c683      1000 MiB             Thu Jun 15 22:34:59 2023  mirror (demoted peer_uuids:[])                                                
    77  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.5d7eb411-baf9-4f18-95e6-bf277e85f060      1000 MiB             Thu Jun 15 22:35:29 2023  mirror (demoted peer_uuids:[])                                                
    84  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.27308766-65cf-457b-a188-f0dc4508e2b0      1000 MiB             Thu Jun 15 22:35:59 2023  mirror (demoted peer_uuids:[])                                                
    91  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.1783f320-4d35-4419-9334-96935ea4067d      1000 MiB             Thu Jun 15 22:36:29 2023  mirror (demoted peer_uuids:[])                                                
    98  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.c379cd15-9a81-430b-9882-a92b7d22cd87      1000 MiB             Thu Jun 15 22:36:59 2023  mirror (demoted peer_uuids:[])                                                
   105  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.e0635a3c-98ce-4521-949e-ca5546ef4dc7      1000 MiB             Thu Jun 15 22:37:29 2023  mirror (demoted peer_uuids:[])                                                
   112  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.12e42cc1-08da-4747-afa2-0b8fff8b0e68      1000 MiB             Thu Jun 15 22:38:00 2023  mirror (demoted peer_uuids:[])                                                
   119  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.6fdda0cd-a363-4d2e-b077-146a7a9cd8f3      1000 MiB             Thu Jun 15 22:38:30 2023  mirror (demoted peer_uuids:[])                                                
   126  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.6a42abbf-75e5-4876-afd5-03f8d53558b9      1000 MiB             Thu Jun 15 22:39:00 2023  mirror (demoted peer_uuids:[])                                                
   133  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.d98f92d6-690f-4d40-ae40-2b6fb73ab513      1000 MiB             Thu Jun 15 22:39:30 2023  mirror (demoted peer_uuids:[])                                                
   136  .mirror.non_primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.1b877383-57dd-47be-a153-0315b0c4b0e6  1000 MiB             Thu Jun 15 22:39:48 2023  mirror (demoted peer_uuids:[] d2562307-bb39-4bd2-b5bd-7d8b137b3d9e:138 copied)
   138  .mirror.primary.373bb9bb-580b-4336-9f82-cb18fb5d96ef.bdb18bd3-35d9-4b97-9963-135a892c21db      1000 MiB             Thu Jun 15 22:39:53 2023  mirror (primary peer_uuids:[4d125e01-7560-4227-af00-3c9d5a414203])

This regression was introduced in: https://github.com/ceph/ceph/commit/c696d24b63492d4be53fec8b7c4da9164f1e2951

Files

reproduce_demote.sh (757 Bytes) reproduce_demote.sh Christopher Hoffman, 06/15/2023 10:49 PM

Related issues 3 (0 open — 3 closed)

Copied to rbd - Backport #62690: quincy: [rbd-mirror] demote snapshot does not get removedResolvedIlya DryomovActions
Copied to rbd - Backport #62691: pacific: [rbd-mirror] demote snapshot does not get removedResolvedIlya DryomovActions
Copied to rbd - Backport #62692: reef: [rbd-mirror] demote snapshot does not get removedResolvedIlya DryomovActions
Actions
Copy link
 #1

Updated by Christopher Hoffman 11 months ago

  • Status changed from New to In Progress
  • Assignee set to Christopher Hoffman
  • Priority changed from Normal to Urgent
Actions
Copy link
 #2

Updated by Christopher Hoffman 11 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 52163
Actions
Copy link
 #3

Updated by Ilya Dryomov 9 months ago

  • Assignee changed from Christopher Hoffman to Ilya Dryomov
  • Pull request ID changed from 52163 to 53251
Actions
Copy link
 #4

Updated by Ilya Dryomov 9 months ago

  • Backport set to pacific,quincy,reef
Actions
Copy link
 #5

Updated by Ilya Dryomov 9 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #6

Updated by Backport Bot 9 months ago

  • Copied to Backport #62690: quincy: [rbd-mirror] demote snapshot does not get removed added
Actions
Copy link
 #7

Updated by Backport Bot 9 months ago

  • Copied to Backport #62691: pacific: [rbd-mirror] demote snapshot does not get removed added
Actions
Copy link
 #8

Updated by Backport Bot 9 months ago

  • Copied to Backport #62692: reef: [rbd-mirror] demote snapshot does not get removed added
Actions
Copy link
 #9

Updated by Backport Bot 9 months ago

  • Tags set to backport_processed
Actions
Copy link
 #10

Updated by Ilya Dryomov 7 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link

Also available in: Atom PDF