Bug #18936
openrgw slo manifest: etag and size should be optional
0%
Description
In debugging a downstream problem with RGW swift SLO, I found the following apparent (?) issues
1. downloading an SLO whose manifest contains paths which do not resolve, possibly unexpected behavior results--for example, if the last of 4 segments in the manifest cannot be resolved, the object can still retrieved by GET--we return just the first 3 segments
the above may be correct behavior, I didn't see guidance in http://docs.openstack.org/developer/swift/overview_large_objects.html
HOWEVER, the swift doc above clearly states that the "etag" and "size_bytes" attributes are optional, so I expect that if the corresponding "path" values are valid, a correct will be assembled; instead, in RGW, the RGW service reports a correct result, but decodes a stored manifest with etag "" and size_bytes 0, total bytes == 0--and the resulting downloads hang until timeout at the client--ditto for swift stat requests.
Files
Updated by Marcus Watts about 7 years ago
For the "hang" problem, I believe that you will find that radosgw didn't issue an "HTTP status" line. I get a hang with both stat and download, and in both cases the output from radosgw is missing the http status line. I'm going to include a copy of "ceph.560" here which will recreate the stat hang (also download hangs but I didn't include an example of that.)
Updated by Matt Benjamin about 7 years ago
to break things down further, here's what happens when just size_bytes or just etag are omitted:
1. omit size_bytes only, for each segment:
[matt@lemon log]# cat nobytes-manifest.json
[
{ "path": "/nflong/sega",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
},
{ "path": "/nflong/segb",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
},
{ "path": "/nflong/segc",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
},
{ "path": "/nflong/segd",
"etag": "558bc3a3928dc6fff9d8e2367985285e"
}
]
command: swift -v stat nflong combo5
result: client hang
2. omit just etag, from each segment:
[matt@lemon log]# cat noetag-manifest.json
[
{ "path": "/nflong/sega",
"size_bytes": 15027082
},
{ "path": "/nflong/segb",
"size_bytes": 15027082
},
{ "path": "/nflong/segc",
"size_bytes": 15027082
},
{ "path": "/nflong/segd",
"size_bytes": 15027082
}
]
action: swift -v stat nflong combo6
result: Object HEAD failed: http://localhost:90/swift/v1/nflong/combo6 412 Precondition Failed
Updated by Matt Benjamin about 7 years ago
Based on review of Swift behavior, agreed behavior for missing etag shall be changed to never report precondition-failed when (optional) etag is not stored for any SLO segment in the manifest.
I'm still root-causing the hang, which turns out happens when the (also optional) size_bytes manifest key is 0 for all segments.
Updated by Matt Benjamin about 7 years ago
I have wip branch which:
1. allows manifeset etag to be optional as required
2. is reorganizing the piecewise read to avoid threading dependency on offsets from the manifest, which should allow offset to be optional as required
Will update when PR is available.
Updated by Matt Benjamin about 7 years ago
will split out fix for optional etag and push PR; fix for optional size is more complex, will continue it in a separate PR
Updated by Matt Benjamin almost 7 years ago
- Subject changed from rgw slo manifest issues to rgw slo manifest: etag and size should be optional
Updated by Matt Benjamin almost 7 years ago
- Priority changed from High to Normal
Updated by Matt Benjamin almost 6 years ago
As of Mimic, an incoming manifest lacking etag and/or size_bytes can be uploaded, but the resulting object cannot be accessed.
Part 1: etag
https://github.com/ceph/ceph/pull/22799
Updated by Matt Benjamin almost 6 years ago
- Status changed from New to In Progress
Updated by Matt Benjamin almost 6 years ago
Updated by Matt Benjamin almost 6 years ago
- Status changed from In Progress to Fix Under Review
- Backport set to mimic, luminous