Ceph » rgw

We see a strange rgw issue: One of our custom applications (written in DotNet, using the MinIO S3 client lib) sometimes receives a broken expiry-date within the x-amz-expiration HTTP header. The bucket in question has a bucket lifecycle policy to expire objects after 7 days.

Here's an example HTTP request and reply:

HEAD https://s3.XXX.XXX.XXX/theBucket/b6755460-6e48-47a5-9695-98f2b12651e7/expiration.txt HTTP/1.1
Host: s3.XXX.XXX.XXX
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20240108T171707Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20240108/europe/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXX
Accept-Encoding: gzip,deflate,br
traceparent: 00-XXXXXXXXXXXXXXXXXXXX-00

HTTP/1.1 200 OK
Content-Length: 10
Last-Modified: Mon, 08 Jan 2024 17:17:05 GMT
Content-Type: text/plain
Accept-Ranges: bytes
x-amz-expiration: expiry-date="Tue, 09 Jan 2023 00:00:00 GMT", rule-id="Delete TUS upload fragments after 7 days and clean-up incomplete uploads" 
x-rgw-object-type: Normal
ETag: "XXX" 
x-amz-request-id: XXX
Date: Mon, 08 Jan 2024 17:17:07 GMT
Connection: Keep-Alive

Here we have multiple issues:

• 09 Jan 2023 was not a Tuesday but a Monday
• An expiry date one year in the past doesn't make any sense in this case

The expiry date can basically be anything, some examples from the same bucket:

• x-amz-expiration: expiry-date="Tue, 31 Jul 2023 00:00:00 GMT", rule-id="Delete TUS upload fragments after 7 days and clean-up incomplete uploads"
• x-amz-expiration: expiry-date="Fri, 16 Jun 2023 00:00:00 GMT", rule-id="Delete TUS upload fragments after 7 days and clean-up incomplete uploads"
• x-amz-expiration: expiry-date="Tue, 21 Jun 2024 00:00:00 GMT", rule-id="Delete TUS upload fragments after 7 days and clean-up incomplete uploads"

But neither does it make sense nor is it a valid date. The issue came up because the MinIO S3 client lib does validate those dates.

The problem is not really reproducible for the same object. For example I checked multiple objects using aws s3api head-object and the x-amz-expiration header looked just nice. But it always comes up once our application issues enough (hundreds or maybe even thousands) requests, maybe in parallel. So right now I'd assume some multithreading issue - maybe the date function used is not thread-safe?

Tested on Ceph 16.2.9 and 16.2.14

Right now the workaround is to disable the bucket lifecycle policy so the x-amz-expiration is missing in the HTTP reply.