Bug #51327
closedFormat of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API)
0%
Description
There is ceph cluster
root@sf-ceph-manager:~# ceph -v
ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
A bucket with object lock/versioning features enabled;
A file with object lock was put in the bucket:
PS> aws2 s3api put-object --bucket vermak-objlock-1 --key file.txt --object-lock-mode COMPLIANCE --object-lock-retain-until "2021-07-07 14:00:00" --endpoint-url "http://172.18.117.25:7480" {
"ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
"VersionId": "VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8"
}
PS> aws2 s3api head-object --bucket vermak-objlock-1 --key file.txt --version-id "VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8" --endpoint-url "http://172.18.117.25:7480" --debug
...
2021-06-23 16:01:56,249 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Length': '0', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Wed, 23 Jun 2021 12:59:24 GMT', 'x-amz-version-id': 'VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8', 'x-rgw-object-type': 'Normal', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'x-amz-object-lock-mode': 'COMPLIANCE', 'x-amz-object-lock-retain-until-date': 'Wed, 07 Jul 2021 14:00:00 GMT', 'x-amz-request-id': 'tx0000000000000000005cc-0060d330c4-3775-default', 'Content-Type': 'binary/octet-stream', 'Date': 'Wed, 23 Jun 2021 13:01:56 GMT', 'Connection': 'Keep-Alive'}
The "x-amz-object-lock-retain-until-date" header is in the wrong format.
The same commands for Amazon S3 returns header that formatted in ISO 8601 standard:
2021-06-23 15:39:37,009 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '3jh2bbm2Zz2LPNhev7h16wxqrFyEqqde9RPAi0YHDRKCqCej1cL3XkarL98FDZaRFQCZqmBMbnk=', 'x-amz-request-id': 'S1F9GCTWM0DN2DC0', 'Date': 'Wed, 23 Jun 2021 12:39:38 GMT', 'Last-Modified': 'Fri, 04 Jun 2021 08:20:02 GMT', 'ETag': '"c81e728d9d4c2f636f067f89cc14862c"', 'x-amz-object-lock-retain-until-date': '2021-06-10T10:43:58Z', 'x-amz-object-lock-mode': 'COMPLIANCE', 'x-amz-object-lock-legal-hold': 'OFF', 'x-amz-version-id': 'cR2SqzzTOvqeTRhTlG2DQw9PKlt5tub1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/octet-stream', 'Content-Length': '1', 'Server': 'AmazonS3'}
Updated by Steven Umbehocker almost 3 years ago
looks like the bug may be in rgw_rest_s3.cc on line 389.
< dump_time_header(s, "x-amz-object-lock-retain-until-date", retention.get_retain_until_date());
That dump_time_header is using this strftime GMT format:
return strftime(timestr, sizeof(timestr),
"%a, %d %b %Y %H:%M:%S %Z", tmp);
Saw there was a method for to_iso_8601 so might be fixable with something like this:
string date = ceph::to_iso_8601(retain_until_date);
dump_header(s, "x-amz-object-lock-retain-until-date", date.c_str());
Updated by Danny Abukalam almost 3 years ago
I've tested that fix on both master and nautilus Steven, and it works.
I've submitted a PR to master here - https://github.com/ceph/ceph/pull/42273.
I believe we should backport this fix once it's been merged.
Updated by Casey Bodley almost 3 years ago
- Status changed from New to Fix Under Review
Updated by Casey Bodley over 2 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 2 years ago
- Copied to Backport #51774: octopus: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) added
Updated by Backport Bot over 2 years ago
- Copied to Backport #51775: pacific: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) added
Updated by Deepika Upadhyay over 2 years ago
Hey Danny!
I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?
Updated by Robert Groenewald over 2 years ago
Deepika Upadhyay wrote:
Hey Danny!
I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?
Is there any progress on this? We have a huge issue with VEEAM right now because of this. It makes Ceph unusable for things like VBR!
Updated by Preben Berg over 2 years ago
I’ve submitted the backport for Octopus: https://github.com/ceph/ceph/pull/43656
Updated by Rolf Knudsen over 2 years ago
Anything but a new octopus release outstandig to get this into production? We're another VBR user which got across this bug as we upgraded to reccomended version. We'd like to skip tape.
Robert Groenewald wrote:
Deepika Upadhyay wrote:
Hey Danny!
I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?
Is there any progress on this? We have a huge issue with VEEAM right now because of this. It makes Ceph unusable for things like VBR!
Updated by David Orman about 2 years ago
We're working on the Pacific backport now (see backport trackers for status/progress, PR is in), and we'll check on the Octopus backport status, as well.
Updated by Justin Goetz about 2 years ago
We were able to build and test the backport here: https://github.com/ceph/ceph/pull/43656 on our Octopus v15.2.14 cluster.
I can confirm the date codes are now being served correctly:
'x-amz-object-lock-retain-until-date': '2022-03-01T14:00:00.000000000Z'
I can confirm after applying the backport, our immutable Veeam buckets began working again for our customers, eliminating the "Unable to convert string with date format..." error that was present in Veeam.
Updated by Yuri Weinstein about 2 years ago
Preben Berg wrote:
I’ve submitted the backport for Octopus: https://github.com/ceph/ceph/pull/43656
merged
Updated by Konstantin Shalygin over 1 year ago
- Status changed from Pending Backport to Resolved
- Tags deleted (
backport_processed)