Bug #55339
closedradosgw rejects some requests without Content-MD5 Header
100%
Description
radosgw server running pacific 16.2.7 on Rocky Linux 8.5 (Green Obsidian).
awscli, running on Rocky Linux 8.5, (Green Obsidian) (different machine than server) This client system is in fips mode. The awscli is 1.22.22 with botocore/1.23.22, the default available with dnf/yum.
The radosgw server is NOT in fips mode
Things like 'aws s3 cp ...' work fine.
However, use of awscli with some s3api calls such as put-bucket-lifecycle-configuration fail with 'InvalidRequest'. If I disable fips mode on the client, then all is well and put-bucket-lifecycle-configuration works fine.
This turns out to be the fact that python's hashlib.md5() is not available when the awscli is running on a fips enabled workstation, and thus awscli doesn't generate a Content-MD5 header. Debug output of awscli shows that Content-MD5: header is not sent when system is in fips mode, but is when fips mode is off.
radosgw seems to need Content-MD5 or it gives 'InvalidRequest'. I have only checked put-bucket-lifecycle-configuration, but I suspect this affects some other s3api calls as well.
In either case (client with fips mode and client without fips mode), header x-amz-content-sha256 header is sent. Perhaps this should be enough for radosgw? (I haven't thought that part through yet). A config option could perhaps be created that disables the need for Content-MD5.
I am aware of https://tracker.ceph.com/issues/53008, but that involves fips mode on the radosgw server itself. The current situation involves use of fips mode on the awscli workstation.
Updated by Matt Benjamin about 2 years ago
Hi Chris,
I suspect you are right that RGW could accept PUT requests without an MD5 header, substituting its own, at least if configured to do so. That said, it doesn't sound to me as if the boto toolchain is behaving correctly in FIPS mode.
Matt
Updated by Casey Bodley almost 2 years ago
- Status changed from New to Triaged
it looks like PutBucketLifecycleConfiguration has some specific logic around this Content-MD5 header, where it's used to verify the checksum of the policy we read:
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_op.cc#L5800
this surprises me - i don't think any other op tries to verify checksums this way. but the amazon docs for PutBucketLifecycleConfiguration do say this header is required:
Important
When making a request using the REST API, you must include the Content-MD5 header.
see https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
so this is indeed a client bug, but i wouldn't object to adding a workaround in rgw. it's just not clear to me why this checksum is necessary for lifecycle
Updated by Casey Bodley almost 2 years ago
- Status changed from Triaged to Fix Under Review
- Tags set to lifecycle fips
- Backport set to octopus pacific quincy
- Pull request ID set to 45990
Updated by Chris Durham almost 2 years ago
Casey Bodley wrote:
it looks like
PutBucketLifecycleConfigurationhas some specific logic around thisContent-MD5header, where it's used to verify the checksum of the policy we read:https://github.com/ceph/ceph/blob/master/src/rgw/rgw_op.cc#L5800
this surprises me - i don't think any other op tries to verify checksums this way. but the amazon docs for PutBucketLifecycleConfiguration do say this header is required:
Important
When making a request using the REST API, you must include the Content-MD5 header.see https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
so this is indeed a client bug, but i wouldn't object to adding a workaround in rgw. it's just not clear to me why this checksum is necessary for lifecycle
I checked with awscli v2. Even in FIPS mode, this version of the awscli works fine. Even in FIPS mode, a Content-MD5 header is sent. The v2 awscli either provides its own version of hashlib.md5() or manipulates the check for fips mode in that routine. This seems odd. awscli v2 is now fully supported by aws and available here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
Updated by J. Eric Ivancich almost 2 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot almost 2 years ago
- Copied to Backport #55504: octopus: radosgw rejects some requests without Content-MD5 Header added
Updated by Backport Bot almost 2 years ago
- Copied to Backport #55505: pacific: radosgw rejects some requests without Content-MD5 Header added
Updated by Backport Bot almost 2 years ago
- Copied to Backport #55506: quincy: radosgw rejects some requests without Content-MD5 Header added
Updated by Backport Bot over 1 year ago
- Tags changed from lifecycle fips to lifecycle fips backport_processed
Updated by Konstantin Shalygin 8 months ago
- Status changed from Pending Backport to Resolved
- % Done changed from 0 to 100