Project

General

Profile

Bug #54592

partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty

Added by Neha Ojha about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

All the OMAP write OPs mark_omap_dirty(), except CEPH_OSD_OP_OMAPRMKEYRANGE. This leads to:

1. incorrectly setting clean_omap
2. not doing omap recovery, when it is actually required
3. PGs in inconsistent state after a scrub


Related issues

Duplicated by RADOS - Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools Duplicate
Copied to RADOS - Backport #55018: quincy: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty Resolved
Copied to RADOS - Backport #55019: octopus: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty Resolved
Copied to RADOS - Backport #55020: pacific: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty Resolved

History

#1 Updated by Neha Ojha about 2 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 45466

#2 Updated by Vikhyat Umrao about 2 years ago

  • Status changed from Fix Under Review to Pending Backport

#3 Updated by Backport Bot about 2 years ago

  • Copied to Backport #55018: quincy: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty added

#4 Updated by Backport Bot about 2 years ago

  • Copied to Backport #55019: octopus: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty added

#5 Updated by Backport Bot about 2 years ago

  • Copied to Backport #55020: pacific: partial recovery: CEPH_OSD_OP_OMAPRMKEYRANGE should mark omap dirty added

#6 Updated by Vikhyat Umrao about 2 years ago

  • Related to Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools added

#7 Updated by Christian Rohmann about 2 years ago

I opened https://tracker.ceph.com/issues/53663 which apparently is expected to be caused by this issue here.
How can we verify that? Our clusters still run on Octopus, what would be the timescale for this backport and a new release?

Considering this issue causes a constant flow of (OMAP) data inconsistencies I'd appreciate to have this fixed asap.

#8 Updated by Neha Ojha almost 2 years ago

  • Related to deleted (Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools)

#9 Updated by Neha Ojha almost 2 years ago

  • Duplicated by Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools added

#10 Updated by Tintu Mathew almost 2 years ago

Verified it downstream by following below steps.

1. Deployed multisite cluster with 16.2.0-152.el8cp version.
2. 2. Added data to both the sites in S3 buckets and the minimal client IO was running.
3. Check for index pool

  1. ceph osd lspools
    1 device_health_metrics
    2 .rgw.root
    3 default.rgw.log
    4 default.rgw.control
    5 default.rgw.meta
    6 primary.rgw.log
    7 primary.rgw.control
    8 primary.rgw.meta
    9 primary.rgw.otp
    10 primary.rgw.buckets.index
    11 primary.rgw.buckets.data

4. Set noout flag
5. Run the deep-scrub for index pool PGs.
6. Identified the objects in index pool

  1. rados -p primary.rgw.buckets.index ls
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.9.8
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.10.0
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.3.8
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.8.3
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.8.5
    .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.10.10

7. Picked an object and identified PG ids.

  1. ceph osd map primary.rgw.buckets.index .dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.10.10
    osdmap e797 pool 'primary.rgw.buckets.index' (10) object '.dir.a6bf6948-e839-4f43-adb7-967a717d7c00.24605.10.10' -> pg 10.395dd0c8 (10.0) -> up ([6,1,0], p6) acting ([6,1,0], p6)

8.Identified on which bucket the object sits.

  1. radosgw-admin metadata list bucket.instance
    [
    "my-bucket-20220408073436-no-4:a6bf6948-e839-4f43-adb7-967a717d7c00.24605.5",
    "my-bucket-20220408073436-no-2:a6bf6948-e839-4f43-adb7-967a717d7c00.24605.3",
    "my-bucket-20220408073436-no-8:a6bf6948-e839-4f43-adb7-967a717d7c00.24605.9",
    "my-bucket-20220408073436-no-9:a6bf6948-e839-4f43-adb7-967a717d7c00.24605.10",
    ]
9. Stopped the secondary OSD on the primary site.
10. Started bilog trimming multiple times while IO running and Scrub running.
  1. radosgw-admin bilog trim --bucket my-bucket-20220408073436-no-9
    11. Started the secondary OSD that was stopped in step 8.
    12. Retriggered the deep-scrub for PGs in index pool.
Srub errors were observed:
  1. ceph health details
    details not valid: details not in detail
    Invalid command: unused arguments: ['details']
    health [detail] : show cluster health
    Error EINVAL: invalid command
    [ceph: root@ceph-pri-tm1-do-not-delete-8pil1m-node1-installer /]# ceph health detail
    HEALTH_ERR noout flag(s) set; 7 scrub errors; Possible data damage: 1 pg inconsistent
    [WRN] OSDMAP_FLAGS: noout flag(s) set
    [ERR] OSD_SCRUB_ERRORS: 7 scrub errors
    [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
    pg 10.0 is active+clean+inconsistent, acting [6,1,0]
13. Repaired the PG
  1. ceph pg repair 10.0

14. Upgraded the cluster with hotfix container image
15. After upgrade bring down secondary osd.
16. Started bilog trimming multiple times while IO running and Scrub running
17. Started the secondary OSD.
18. Retriggered the deep-scrub for PGs in index pool.

The scrub issues were not observed.

#12 Updated by Neha Ojha almost 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF