Project

General

Profile

Bug #18936

rgw slo manifest: etag and size should be optional

Added by Matt Benjamin over 2 years ago. Updated about 1 year ago.

Status:
Need Review
Priority:
Normal
Assignee:
Target version:
-
Start date:
02/14/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
mimic, luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

In debugging a downstream problem with RGW swift SLO, I found the following apparent (?) issues

1. downloading an SLO whose manifest contains paths which do not resolve, possibly unexpected behavior results--for example, if the last of 4 segments in the manifest cannot be resolved, the object can still retrieved by GET--we return just the first 3 segments

the above may be correct behavior, I didn't see guidance in http://docs.openstack.org/developer/swift/overview_large_objects.html

HOWEVER, the swift doc above clearly states that the "etag" and "size_bytes" attributes are optional, so I expect that if the corresponding "path" values are valid, a correct will be assembled; instead, in RGW, the RGW service reports a correct result, but decodes a stored manifest with etag "" and size_bytes 0, total bytes == 0--and the resulting downloads hang until timeout at the client--ditto for swift stat requests.

ceph.560 - mdw's cut & paste SLO script (2.4 KB) Marcus Watts, 02/15/2017 12:30 AM

History

#1 Updated by Marcus Watts over 2 years ago

For the "hang" problem, I believe that you will find that radosgw didn't issue an "HTTP status" line. I get a hang with both stat and download, and in both cases the output from radosgw is missing the http status line. I'm going to include a copy of "ceph.560" here which will recreate the stat hang (also download hangs but I didn't include an example of that.)

#2 Updated by Matt Benjamin over 2 years ago

to break things down further, here's what happens when just size_bytes or just etag are omitted:

1. omit size_bytes only, for each segment:
[matt@lemon log]# cat nobytes-manifest.json
[ { "path": "/nflong/sega",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
}, { "path": "/nflong/segb",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
}, { "path": "/nflong/segc",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
}, { "path": "/nflong/segd",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
}
]

command: swift -v stat nflong combo5
result: client hang

2. omit just etag, from each segment:
[matt@lemon log]# cat noetag-manifest.json
[ { "path": "/nflong/sega",
"size_bytes": 15027082
}, { "path": "/nflong/segb",
"size_bytes": 15027082
}, { "path": "/nflong/segc",
"size_bytes": 15027082
}, { "path": "/nflong/segd",
"size_bytes": 15027082
}
]

action: swift -v stat nflong combo6
result: Object HEAD failed: http://localhost:90/swift/v1/nflong/combo6 412 Precondition Failed

#3 Updated by Matt Benjamin over 2 years ago

Based on review of Swift behavior, agreed behavior for missing etag shall be changed to never report precondition-failed when (optional) etag is not stored for any SLO segment in the manifest.

I'm still root-causing the hang, which turns out happens when the (also optional) size_bytes manifest key is 0 for all segments.

#4 Updated by Yehuda Sadeh over 2 years ago

  • Priority changed from Normal to High

#5 Updated by Matt Benjamin over 2 years ago

I have wip branch which:
1. allows manifeset etag to be optional as required
2. is reorganizing the piecewise read to avoid threading dependency on offsets from the manifest, which should allow offset to be optional as required

Will update when PR is available.

#6 Updated by Matt Benjamin over 2 years ago

will split out fix for optional etag and push PR; fix for optional size is more complex, will continue it in a separate PR

#7 Updated by Matt Benjamin about 2 years ago

  • Subject changed from rgw slo manifest issues to rgw slo manifest: etag and size should be optional

#8 Updated by Matt Benjamin about 2 years ago

  • Priority changed from High to Normal

#9 Updated by Matt Benjamin about 1 year ago

As of Mimic, an incoming manifest lacking etag and/or size_bytes can be uploaded, but the resulting object cannot be accessed.

Part 1: etag
https://github.com/ceph/ceph/pull/22799

#10 Updated by Matt Benjamin about 1 year ago

  • Status changed from New to In Progress

#12 Updated by Matt Benjamin about 1 year ago

  • Status changed from In Progress to Need Review
  • Backport set to mimic, luminous

Also available in: Atom PDF