Project

General

Profile

Bug #16435

COPY broke multipart files uploaded under dumpling

Added by Robin Johnson almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
06/22/2016
Due date:
% Done:

0%

Source:
other
Tags:
hammer, dumpling, rgw, multipart
Backport:
jewel, hammer
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw

Description

A user reported that some files in their bucket, uploaded ~2 years ago under dumpling, have stopped working after they were moved around recently.

All of the files under "home/fvault/Lastest/" were moved to "home/fvault/Lastest/20g/", and now they all don't work anymore.

A HEAD on the files returns the correct metadata and response code 200, but a GET operation and all other operations [that access the objects] I've tested (including rename/move) returns 404.

I have run:
- bucket check --fix
- bucket check --fix --check-head-obj-locator
But it did not fix the issue.

The rgw-admin object stat output on one of the broken objects does show the older manifest->objs structure, and the first item in that list contains the old filename, isn't in RADOS anymore.

Attached is rgw-admin object stat output, plus rados stat output for each of the parts.

I've censored the following information in the attachments.
- IPs
- Bucket name
- Actual filename

Required fix parts:
- codebase needs to correctly handle the move
- need a fixup tool to detect and fix all broken manifests for this, may be VERY hard depending what the rename was.

rgw-object-stat.dho1444.txt View - radosgw-admin object stat (4.05 KB) Robin Johnson, 06/22/2016 10:24 PM

rados-objects.stat.dho1444.txt View - rados stat (365 Bytes) Robin Johnson, 06/22/2016 10:24 PM

rgw-log.dho1444.txt View - RGW logs for the object (HEAD+GET) (2.93 KB) Robin Johnson, 06/22/2016 10:24 PM


Related issues

Copied to rgw - Backport #17122: jewel: COPY broke multipart files uploaded under dumpling Resolved
Copied to rgw - Backport #17123: hammer: COPY broke multipart files uploaded under dumpling Resolved

History

#1 Updated by Robin Johnson almost 2 years ago

  • Subject changed from multipart files uploaded under dumpling broken in hammer, after moving folders to COPY broke multipart files uploaded under dumpling
  • Release set to infernalis
  • Release set to jewel

Per IRC discussion with Yehuda, the COPY operation has broken the manifest.

Things to fix:
  1. Fix COPY to not break the manifest
  2. Add Fix tool to fix broken manifests
  3. (maybe) Fix GET to handle broken manifests transparently

Also it would seem that this does affect Jewel environments with the old Dumpling data as well.

#2 Updated by Orit Wasserman almost 2 years ago

  • Assignee set to Zheng Yan

#3 Updated by Orit Wasserman almost 2 years ago

  • Assignee changed from Zheng Yan to Yehuda Sadeh

#4 Updated by Yehuda Sadeh over 1 year ago

  • Status changed from New to Pending Backport
  • Backport set to jewel, hammer

#5 Updated by Loic Dachary over 1 year ago

  • Copied to Backport #17122: jewel: COPY broke multipart files uploaded under dumpling added

#6 Updated by Loic Dachary over 1 year ago

  • Copied to Backport #17123: hammer: COPY broke multipart files uploaded under dumpling added

#7 Updated by Loic Dachary over 1 year ago

  • Target version deleted (v0.94.8)

#8 Updated by Nathan Cutler over 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF