Bug #58908
openKeystone EC2 auth does not support STREAMING-AWS4-HMAC-SHA256-PAYLOAD
0%
Description
Tested on current Pacific (16.2.11), with Keystone Xena, the same test works on local rgw user.
The test is to use any client capable of chunked upload, warp form MinIO here. On large object chunked upload it refuses with 501 Not Implemented response when keystone access key is used, while using local rgw user it passes. This happens of course only with s3v4 signature on STREAMING-AWS4-HMAC-SHA256-PAYLOAD transfer.
Debug log
2023-03-03T14:29:28.717+0000 7fb6635d6700 1 beast: 0x7fb80c4846e0: 10.21.1.11 - 30200495337a417aa322288e36a67d9d [03/Mar/2023:14:29:28.713 +0000] "GET /warp-benchmark-bucket/?delimiter=&encoding-type=url
&fetch-owner=true&list-type=2&prefix= HTTP/1.1" 200 295 - "MinIO (linux; amd64) minio-go/v7.0.47 warp/0.6.6" - latency=0.004000011s
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 CONTENT_LENGTH=17934
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 CONTENT_TYPE=application/octet-stream
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-
date;x-amz-decoded-content-length,Signature=15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_CONNECTION=close
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_HOST=object.az1.devtest.local
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_USER_AGENT=MinIO (linux; amd64) minio-go/v7.0.47 warp/0.6.6
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_VERSION=1.1
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_CONTENT_SHA256=STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_DATE=20230303T142928Z
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_DECODED_CONTENT_LENGTH=17759
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_FORWARDED_FOR=10.21.1.11
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REMOTE_ADDR=10.21.1.11
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REQUEST_METHOD=PUT
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REQUEST_URI=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 SCRIPT_URI=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 SERVER_PORT=8080
2023-03-03T14:29:28.721+0000 7fb716f3d700 1 ====== starting new request req=0x7fb80c4846e0 =====
2023-03-03T14:29:28.721+0000 7fb716f3d700 2 req 13682662579721275793 0.000000000s initializing for trans_id = tx00000bde2951f55d10991-0064020448-2d61e9-az1
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s rgw api priority: s3=8 s3website=7
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s host=object.az1.devtest.local
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_ur
i=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_CONTENT_SHA256
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_DATE
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_DECODED_CONTENT_LENGTH
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-date:20230303T142928Z
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-decoded-content-length:17759
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s handler=22RGWHandler_REST_Obj_S3
2023-03-03T14:29:28.721+0000 7fb716f3d700 2 req 13682662579721275793 0.000000000s getting op 1
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj scheduling with throttler client=2 cost=1
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj op=21RGWPutObj_ObjStore_S3
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj verifying requester
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s payload request hash = STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request = PUT
/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s date_k = 09612b40b246d9e3ad696eb2db88b68fb33bb1df778fa95a169f3c7e497f7080
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s region_k = 42ab646d3cc2db152885e329f805189d27eef5974f92f5d832f69b598ae421a3
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s service_k = aa55192295115a5c79dc3d5f0bed60f6a39dfdc3bfa37b469b7b0e35e708f3b6
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s signing_k = 3446bc83dab69902f2f197250c08f4b849a5d48e265c62b55295dff3ae9601d6
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s generated signature = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj s3 keystone: validated token: stor_test:stor_test expires: 1677896734
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::keystone::EC2Engine denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s payload request hash = STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request = PUT
/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj get_system_obj_state: rctx=0x7fb80c482c68 obj=az1.rgw.meta:users.keys:798869c7266e4965aa5151f9a1924083 state=0
x7fb800253d00 s->prefetch_data=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj cache get: name=az1.rgw.meta+users.keys+798869c7266e4965aa5151f9a1924083 : hit (negative entry)
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj error reading user info, uid=798869c7266e4965aa5151f9a1924083 can't authenticate
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::LocalEngine denied with reason=-2028
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s payload request hash = STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request = PUT
/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759
host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj get_system_obj_state: rctx=0x7fb80c482c68 obj=az1.rgw.meta:users.keys:798869c7266e4965aa5151f9a1924083 state=0
x7fb800253d00 s->prefetch_data=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj cache get: name=az1.rgw.meta+users.keys+798869c7266e4965aa5151f9a1924083 : hit (negative entry)
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj error reading user info, uid=798869c7266e4965aa5151f9a1924083 can't authenticate
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::LocalEngine denied with reason=-2028
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj Failed the auth strategy, reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 failed to authorize request
2023-03-03T14:29:28.725+0000 7fb716f3d700 1 req 13682662579721275793 0.004000011s op->ERRORHANDLER: err_no=-2201 new_err_no=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj op status=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj http status=501
2023-03-03T14:29:28.725+0000 7fb716f3d700 1 ====== req done req=0x7fb80c4846e0 op status=0 http_status=501 latency=0.004000011s ======
(in above real hostname and region name were obfuscated)
The correct behaviour is to accept PUT and continue upload. This works with a local rgw user or with signature s3v2. Other operations/methods like GET/STAT/DELETE works fine with s3v4 and Keystone.