Project

General

Profile

Bug #51327

Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API)

Added by Vladimir Ermak over 1 year ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

There is ceph cluster

root@sf-ceph-manager:~# ceph -v
ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)

A bucket with object lock/versioning features enabled;

A file with object lock was put in the bucket:


PS> aws2 s3api put-object --bucket vermak-objlock-1 --key file.txt --object-lock-mode COMPLIANCE --object-lock-retain-until "2021-07-07 14:00:00" --endpoint-url "http://172.18.117.25:7480" {
"ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
"VersionId": "VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8"
}

PS> aws2 s3api head-object --bucket vermak-objlock-1 --key file.txt --version-id "VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8" --endpoint-url "http://172.18.117.25:7480" --debug
...
2021-06-23 16:01:56,249 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Length': '0', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Wed, 23 Jun 2021 12:59:24 GMT', 'x-amz-version-id': 'VqY2-Go7ZdC8-c6G8OKDyDNI82Gg4S8', 'x-rgw-object-type': 'Normal', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'x-amz-object-lock-mode': 'COMPLIANCE', 'x-amz-object-lock-retain-until-date': 'Wed, 07 Jul 2021 14:00:00 GMT', 'x-amz-request-id': 'tx0000000000000000005cc-0060d330c4-3775-default', 'Content-Type': 'binary/octet-stream', 'Date': 'Wed, 23 Jun 2021 13:01:56 GMT', 'Connection': 'Keep-Alive'}

The "x-amz-object-lock-retain-until-date" header is in the wrong format.

The same commands for Amazon S3 returns header that formatted in ISO 8601 standard:

2021-06-23 15:39:37,009 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '3jh2bbm2Zz2LPNhev7h16wxqrFyEqqde9RPAi0YHDRKCqCej1cL3XkarL98FDZaRFQCZqmBMbnk=', 'x-amz-request-id': 'S1F9GCTWM0DN2DC0', 'Date': 'Wed, 23 Jun 2021 12:39:38 GMT', 'Last-Modified': 'Fri, 04 Jun 2021 08:20:02 GMT', 'ETag': '"c81e728d9d4c2f636f067f89cc14862c"', 'x-amz-object-lock-retain-until-date': '2021-06-10T10:43:58Z', 'x-amz-object-lock-mode': 'COMPLIANCE', 'x-amz-object-lock-legal-hold': 'OFF', 'x-amz-version-id': 'cR2SqzzTOvqeTRhTlG2DQw9PKlt5tub1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/octet-stream', 'Content-Length': '1', 'Server': 'AmazonS3'}


Related issues

Copied to Ceph - Backport #51774: octopus: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) Resolved
Copied to Ceph - Backport #51775: pacific: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) Resolved

History

#1 Updated by Steven Umbehocker over 1 year ago

looks like the bug may be in rgw_rest_s3.cc on line 389.

< dump_time_header(s, "x-amz-object-lock-retain-until-date", retention.get_retain_until_date());

That dump_time_header is using this strftime GMT format:

return strftime(timestr, sizeof(timestr),
"%a, %d %b %Y %H:%M:%S %Z", tmp);

Saw there was a method for to_iso_8601 so might be fixable with something like this:

string date = ceph::to_iso_8601(retain_until_date);

dump_header(s, "x-amz-object-lock-retain-until-date", date.c_str());

#2 Updated by Matt Benjamin over 1 year ago

  • Assignee set to Matt Benjamin

#3 Updated by Danny Abukalam about 1 year ago

I've tested that fix on both master and nautilus Steven, and it works.

I've submitted a PR to master here - https://github.com/ceph/ceph/pull/42273.

I believe we should backport this fix once it's been merged.

#4 Updated by Matt Benjamin about 1 year ago

  • Pull request ID set to 42273

#5 Updated by Casey Bodley about 1 year ago

  • Backport set to octopus pacific

#6 Updated by Casey Bodley about 1 year ago

  • Status changed from New to Fix Under Review

#7 Updated by Casey Bodley about 1 year ago

  • Status changed from Fix Under Review to Pending Backport

#8 Updated by Backport Bot about 1 year ago

  • Copied to Backport #51774: octopus: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) added

#9 Updated by Backport Bot about 1 year ago

  • Copied to Backport #51775: pacific: Format of date/time header "x-amz-object-lock-retain-until-date" is incorrect (does not meet the S3 API) added

#10 Updated by Deepika Upadhyay about 1 year ago

Hey Danny!

I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?

#11 Updated by Robert Groenewald 11 months ago

Deepika Upadhyay wrote:

Hey Danny!

I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?

Is there any progress on this? We have a huge issue with VEEAM right now because of this. It makes Ceph unusable for things like VBR!

#12 Updated by Preben Berg 11 months ago

I’ve submitted the backport for Octopus: https://github.com/ceph/ceph/pull/43656

#13 Updated by Greg Farnum 11 months ago

  • Project changed from Ceph to rgw

#14 Updated by Rolf Knudsen 9 months ago

Anything but a new octopus release outstandig to get this into production? We're another VBR user which got across this bug as we upgraded to reccomended version. We'd like to skip tape.

Robert Groenewald wrote:

Deepika Upadhyay wrote:

Hey Danny!

I see already existing tracker issues for backports, do you want to create backports for them: https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#opening-a-backport-pr ?

Is there any progress on this? We have a huge issue with VEEAM right now because of this. It makes Ceph unusable for things like VBR!

#15 Updated by David Orman 8 months ago

We're working on the Pacific backport now (see backport trackers for status/progress, PR is in), and we'll check on the Octopus backport status, as well.

#16 Updated by Justin Goetz 8 months ago

We were able to build and test the backport here: https://github.com/ceph/ceph/pull/43656 on our Octopus v15.2.14 cluster.

I can confirm the date codes are now being served correctly:

'x-amz-object-lock-retain-until-date': '2022-03-01T14:00:00.000000000Z'

I can confirm after applying the backport, our immutable Veeam buckets began working again for our customers, eliminating the "Unable to convert string with date format..." error that was present in Veeam.

#17 Updated by Yuri Weinstein 7 months ago

Preben Berg wrote:

I’ve submitted the backport for Octopus: https://github.com/ceph/ceph/pull/43656

merged

#18 Updated by Backport Bot about 2 months ago

  • Tags set to backport_processed

#19 Updated by Konstantin Shalygin about 1 month ago

  • Status changed from Pending Backport to Resolved
  • Tags deleted (backport_processed)

Also available in: Atom PDF