Project

General

Profile

Actions

Bug #58034

closed

RGW misplaces index entries after dynamically resharding bucket

Added by J. Eric Ivancich over 1 year ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Target version:
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
quincy,pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When RGW reshards buckets with ~250k index entries*, I've noticed some s3:PutObject requests that return 200 end up with index entries under the old index shard oids, presumably recreated after dynamically resharding deleted the original shards. These objects can be retrieved via s3:GetObject, since that doesn't typically look for an index entry, but s3:ListObjects doesn't show any information about the successfully written object.

This does not happen reliably. I've been able to recreate this in a staging environment in 1-4 of every 30 buckets, with and without rgw caching enabled on the rgw nodes serving the s3:PutObject requests. mtime on the misplaced index entries shows a burst of writes (all within a few seconds) going to the wrong index after resharding completes, with the majority of writes going to the correct index. This burst can happen multiple times after resharding, from 20s after resharding completes to 12m. I haven't spotted any useful error debug logging in the client logs, but have also not turned up logging very high at all, in part due the volume of requests required for recreating this issue. Let me know if there's any debug information that would be useful. Thanks for reading!

  • I don't think 250k is significant in any way, it's just the threshold at which we tend to reshard the most buckets.

Related issues 3 (0 open3 closed)

Copied from rgw - Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucketResolvedJ. Eric Ivancich

Actions
Copied to rgw - Backport #58170: pacific: RGW misplaces index entries after dynamically resharding bucketDuplicateJ. Eric IvancichActions
Copied to rgw - Backport #58171: quincy: RGW misplaces index entries after dynamically resharding bucketResolvedJ. Eric IvancichActions
Actions #1

Updated by J. Eric Ivancich over 1 year ago

  • Copied from Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket added
Actions #2

Updated by J. Eric Ivancich over 1 year ago

  • Pull request ID set to 48899
Actions #3

Updated by Casey Bodley over 1 year ago

  • Status changed from New to In Progress
Actions #4

Updated by J. Eric Ivancich over 1 year ago

  • Status changed from In Progress to Pending Backport
  • Backport set to quincy,pacific
Actions #5

Updated by Backport Bot over 1 year ago

  • Copied to Backport #58170: pacific: RGW misplaces index entries after dynamically resharding bucket added
Actions #6

Updated by Backport Bot over 1 year ago

  • Copied to Backport #58171: quincy: RGW misplaces index entries after dynamically resharding bucket added
Actions #7

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions #8

Updated by Konstantin Shalygin 7 months ago

  • Status changed from Pending Backport to Resolved
  • Source set to Community (user)
Actions

Also available in: Atom PDF