Project

General

Profile

Bug #58034

RGW misplaces index entries after dynamically resharding bucket

Added by J. Eric Ivancich 3 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Normal
Target version:
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy,pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When RGW reshards buckets with ~250k index entries*, I've noticed some s3:PutObject requests that return 200 end up with index entries under the old index shard oids, presumably recreated after dynamically resharding deleted the original shards. These objects can be retrieved via s3:GetObject, since that doesn't typically look for an index entry, but s3:ListObjects doesn't show any information about the successfully written object.

This does not happen reliably. I've been able to recreate this in a staging environment in 1-4 of every 30 buckets, with and without rgw caching enabled on the rgw nodes serving the s3:PutObject requests. mtime on the misplaced index entries shows a burst of writes (all within a few seconds) going to the wrong index after resharding completes, with the majority of writes going to the correct index. This burst can happen multiple times after resharding, from 20s after resharding completes to 12m. I haven't spotted any useful error debug logging in the client logs, but have also not turned up logging very high at all, in part due the volume of requests required for recreating this issue. Let me know if there's any debug information that would be useful. Thanks for reading!

  • I don't think 250k is significant in any way, it's just the threshold at which we tend to reshard the most buckets.

Related issues

Copied from rgw - Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket Resolved
Copied to rgw - Backport #58170: pacific: RGW misplaces index entries after dynamically resharding bucket Duplicate
Copied to rgw - Backport #58171: quincy: RGW misplaces index entries after dynamically resharding bucket In Progress

History

#1 Updated by J. Eric Ivancich 3 months ago

  • Copied from Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket added

#2 Updated by J. Eric Ivancich 3 months ago

  • Pull request ID set to 48899

#3 Updated by Casey Bodley 3 months ago

  • Status changed from New to In Progress

#4 Updated by J. Eric Ivancich about 2 months ago

  • Status changed from In Progress to Pending Backport
  • Backport set to quincy,pacific

#5 Updated by Backport Bot about 2 months ago

  • Copied to Backport #58170: pacific: RGW misplaces index entries after dynamically resharding bucket added

#6 Updated by Backport Bot about 2 months ago

  • Copied to Backport #58171: quincy: RGW misplaces index entries after dynamically resharding bucket added

#7 Updated by Backport Bot about 2 months ago

  • Tags set to backport_processed

Also available in: Atom PDF