Project

General

Profile

Bug #16435

COPY broke multipart files uploaded under dumpling

Added by Robin Johnson 11 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
06/22/2016
Due date:
% Done:

0%

Source:
other
Tags:
hammer, dumpling, rgw, multipart
Backport:
jewel, hammer
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Release:
hammer, infernalis, jewel
Needs Doc:
No

Description

A user reported that some files in their bucket, uploaded ~2 years ago under dumpling, have stopped working after they were moved around recently.

All of the files under "home/fvault/Lastest/" were moved to "home/fvault/Lastest/20g/", and now they all don't work anymore.

A HEAD on the files returns the correct metadata and response code 200, but a GET operation and all other operations [that access the objects] I've tested (including rename/move) returns 404.

I have run:
- bucket check --fix
- bucket check --fix --check-head-obj-locator
But it did not fix the issue.

The rgw-admin object stat output on one of the broken objects does show the older manifest->objs structure, and the first item in that list contains the old filename, isn't in RADOS anymore.

Attached is rgw-admin object stat output, plus rados stat output for each of the parts.

I've censored the following information in the attachments.
- IPs
- Bucket name
- Actual filename

Required fix parts:
- codebase needs to correctly handle the move
- need a fixup tool to detect and fix all broken manifests for this, may be VERY hard depending what the rename was.

rgw-object-stat.dho1444.txt View - radosgw-admin object stat (4.05 KB) Robin Johnson, 06/22/2016 10:24 PM

rados-objects.stat.dho1444.txt View - rados stat (365 Bytes) Robin Johnson, 06/22/2016 10:24 PM

rgw-log.dho1444.txt View - RGW logs for the object (HEAD+GET) (2.93 KB) Robin Johnson, 06/22/2016 10:24 PM


Related issues

Copied to Backport #17122: jewel: COPY broke multipart files uploaded under dumpling Resolved
Copied to Backport #17123: hammer: COPY broke multipart files uploaded under dumpling Resolved

History

#1 Updated by Robin Johnson 11 months ago

  • Subject changed from multipart files uploaded under dumpling broken in hammer, after moving folders to COPY broke multipart files uploaded under dumpling
  • Release infernalis, jewel added

Per IRC discussion with Yehuda, the COPY operation has broken the manifest.

Things to fix:
  1. Fix COPY to not break the manifest
  2. Add Fix tool to fix broken manifests
  3. (maybe) Fix GET to handle broken manifests transparently

Also it would seem that this does affect Jewel environments with the old Dumpling data as well.

#2 Updated by Orit Wasserman 11 months ago

  • Assignee set to Zheng Yan

#3 Updated by Orit Wasserman 11 months ago

  • Assignee changed from Zheng Yan to Yehuda Sadeh

#4 Updated by Yehuda Sadeh 9 months ago

  • Status changed from New to Pending Backport
  • Backport set to jewel, hammer

#5 Updated by Loic Dachary 9 months ago

  • Copied to Backport #17122: jewel: COPY broke multipart files uploaded under dumpling added

#6 Updated by Loic Dachary 9 months ago

  • Copied to Backport #17123: hammer: COPY broke multipart files uploaded under dumpling added

#7 Updated by Loic Dachary 9 months ago

  • Target version deleted (v0.94.8)

#8 Updated by Nathan Cutler 6 months ago

  • Status changed from Pending Backport to Resolved
  • Needs Doc set to No

Also available in: Atom PDF