Actions
Bug #63953
closedInvalidRequest when using SSE-C
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I try to encrypt object using SSE-C when uploading it to CEPH S3 endpoint.
I do it like:
./bin/mc cp ~/tmp/config.js tester1/test/config.js --encrypt-key "tester1/test/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo=" --debug
but I get InvalidRequest response.
mc: <DEBUG> GET /test/?location= HTTP/1.1 Host: myserver.internet.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-29T20-15-29Z Accept-Encoding: zstd,gzip Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240105/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: UNSIGNED-PAYLOAD X-Amz-Date: 20240105T151958Z mc: <DEBUG> HTTP/1.1 200 OK Content-Length: 134 Access-Control-Allow-Headers: authorization,x-amz-content-sha256,x-amz-date,content-md5,content-type Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: ETag Connection: keep-alive Date: Fri, 05 Jan 2024 15:19:58 GMT Server: nginx X-Amz-Request-Id: tx000004bd55e141ad2c229-0065981e1e-1c16607-default mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: NL mc: <DEBUG> >> Organization: GEANT Vereniging mc: <DEBUG> >> Expires: 2024-02-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: The USERTRUST Network mc: <DEBUG> >> Expires: 2033-05-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> Response Time: 41.726957ms mc: <DEBUG> GET /test/?object-lock= HTTP/1.1 Host: myserver.internet.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-29T20-15-29Z Accept-Encoding: zstd,gzip Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240105/default/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20240105T151958Z mc: <DEBUG> HTTP/1.1 404 Not Found Content-Length: 263 Accept-Ranges: bytes Access-Control-Allow-Headers: authorization,x-amz-content-sha256,x-amz-date,content-md5,content-type Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: ETag Connection: keep-alive Content-Type: application/xml Date: Fri, 05 Jan 2024 15:19:58 GMT Server: nginx X-Amz-Request-Id: tx000001e1a60a9b3174fab-0065981e1e-1c16607-default <?xml version="1.0" encoding="UTF-8"?><Error><Code>ObjectLockConfigurationNotFoundError</Code><Message></Message><BucketName>test</BucketName><RequestId>tx000001e1a60a9b3174fab-0065981e1e-1c16607-default</RequestId><HostId>1c16607-default-default</HostId></Error>mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: NL mc: <DEBUG> >> Organization: GEANT Vereniging mc: <DEBUG> >> Expires: 2024-02-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: The USERTRUST Network mc: <DEBUG> >> Expires: 2033-05-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> Response Time: 47.652101ms mc: <DEBUG> HEAD /test/config.js HTTP/1.1 Host: myserver.internet.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-29T20-15-29Z Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240105/default/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20240105T151958Z X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256 X-Amz-Server-Side-Encryption-Customer-Key: c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo= X-Amz-Server-Side-Encryption-Customer-Key-Md5: FU3M6aHsCmEsXXCg4gcT2A== mc: <DEBUG> HTTP/1.1 404 Not Found Content-Length: 236 Accept-Ranges: bytes Access-Control-Allow-Headers: authorization,x-amz-content-sha256,x-amz-date,content-md5,content-type Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: ETag Connection: keep-alive Content-Type: application/xml Date: Fri, 05 Jan 2024 15:19:58 GMT Server: nginx X-Amz-Request-Id: tx000001deef96270b1de53-0065981e1e-1c16607-default mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: NL mc: <DEBUG> >> Organization: GEANT Vereniging mc: <DEBUG> >> Expires: 2024-02-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: The USERTRUST Network mc: <DEBUG> >> Expires: 2033-05-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> Response Time: 8.724017ms 0 B / ? ━┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉━━mc: <DEBUG> GET /test/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=config.js%2F HTTP/1.1 Host: myserver.internet.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-29T20-15-29Z Accept-Encoding: zstd,gzip Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240105/default/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20240105T151958Z mc: <DEBUG> HTTP/1.1 200 OK Transfer-Encoding: chunked Access-Control-Allow-Headers: authorization,x-amz-content-sha256,x-amz-date,content-md5,content-type Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: ETag Connection: keep-alive Content-Type: application/xml Date: Fri, 05 Jan 2024 15:19:58 GMT Server: nginx X-Amz-Request-Id: tx00000df85fe212dca0ff3-0065981e1e-1c16607-default mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: NL mc: <DEBUG> >> Organization: GEANT Vereniging mc: <DEBUG> >> Expires: 2024-02-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: The USERTRUST Network mc: <DEBUG> >> Expires: 2033-05-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> Response Time: 49.626667ms mc: <DEBUG> PUT /test/config.js HTTP/1.1 Host: myserver.internet.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-29T20-15-29Z Content-Length: 162 Accept-Encoding: zstd,gzip Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240105/default/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5, Signature=**REDACTED** Content-Type: text/javascript X-Amz-Content-Sha256: UNSIGNED-PAYLOAD X-Amz-Date: 20240105T151958Z X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256 X-Amz-Server-Side-Encryption-Customer-Key: c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo= X-Amz-Server-Side-Encryption-Customer-Key-Md5: FU3M6aHsCmEsXXCg4gcT2A== mc: <DEBUG> HTTP/1.1 400 Bad Request Content-Length: 241 Accept-Ranges: bytes Access-Control-Allow-Headers: authorization,x-amz-content-sha256,x-amz-date,content-md5,content-type Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: ETag Connection: keep-alive Content-Type: application/xml Date: Fri, 05 Jan 2024 15:19:58 GMT Server: nginx X-Amz-Request-Id: tx0000005c10aa8724195c2-0065981e1e-1c16607-default <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message></Message><BucketName>test</BucketName><RequestId>tx0000005c10aa8724195c2-0065981e1e-1c16607-default</RequestId><HostId>1c16607-default-default</HostId></Error>mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: NL mc: <DEBUG> >> Organization: GEANT Vereniging mc: <DEBUG> >> Expires: 2024-02-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: The USERTRUST Network mc: <DEBUG> >> Expires: 2033-05-01 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: GB mc: <DEBUG> >> Organization: Comodo CA Limited mc: <DEBUG> >> Expires: 2028-12-31 23:59:59 +0000 UTC mc: <DEBUG> Response Time: 47.642057ms mc: <ERROR> Failed to copy `/home/username/tmp/config.js`. Error response code InvalidRequest. (3) cp-main.go:610 cmd.doCopySession(..) Tags: [/home/username/tmp/config.js] (2) common-methods.go:607 cmd.uploadSourceToTargetURL(..) Tags: [/home/username/tmp/config.js] (1) common-methods.go:337 cmd.putTargetStream(..) Tags: [tester1, https://myserver.internet.com/test/config.js] (0) client-s3.go:1214 cmd.(*S3Client).Put(..) Release-Tag:RELEASE.2023-12-29T20-15-29Z | Commit:5386533bb912 | Host:buro | OS:linux | Arch:amd64 | Lang:go1.21.5 | Mem:5.2 MiB/20 MiB | Heap:5.2 MiB/11 MiB
Any ideas why?
Updated by Casey Bodley 4 months ago
have you learned anything from the radosgw log? hopefully with debug_rgw=20 you can find a reason for the 400 error
Updated by Rok Jaklic 4 months ago
Casey Bodley wrote:
have you learned anything from the radosgw log? hopefully with debug_rgw=20 you can find a reason for the 400 error
In logs I got
2024-01-10T12:58:55.705+0100 7f928117e700 5 req 4401260719049339428 0.001000008s ERROR: Insecure request, rgw_crypt_require_ssl is set
We are using nginx in front of ceph rgw nodes. On public IP we require ssl, but on ceph we do not. We've set rgw_crypt_require_ssl to false and object encryption using SSE-C now works.
Thx.
Updated by Casey Bodley 4 months ago
- Status changed from New to Resolved
Rok Jaklic wrote:
Casey Bodley wrote:
have you learned anything from the radosgw log? hopefully with debug_rgw=20 you can find a reason for the 400 error
In logs I got
[...]We are using nginx in front of ceph rgw nodes. On public IP we require ssl, but on ceph we do not. We've set rgw_crypt_require_ssl to false and object encryption using SSE-C now works.
makes sense, thanks for following up!
note that there's a separate option rgw_trust_forwarded_https=true1 that you might consider instead of rgw_crypt_require_ssl=false
[1] https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_trust_forwarded_https
Actions