Bug #58908: Keystone EC2 auth does not support STREAMING-AWS4-HMAC-SHA256-PAYLOAD - rgw - Ceph

Actions

Copy link

Bug #58908

open

Keystone EC2 auth does not support STREAMING-AWS4-HMAC-SHA256-PAYLOAD

Added by Pawel Stefanski about 1 year ago. Updated 10 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Casey Bodley

Target version:

% Done:

Source:

Community (user)

Tags:

rgw, s3, keystone backport_processed

Backport:

pacific quincy reef

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v16.2.11

ceph-qa-suite:

Pull request ID:

50550

Crash signature (v1):

Crash signature (v2):

Description

Tested on current Pacific (16.2.11), with Keystone Xena, the same test works on local rgw user.

The test is to use any client capable of chunked upload, warp form MinIO here. On large object chunked upload it refuses with 501 Not Implemented response when keystone access key is used, while using local rgw user it passes. This happens of course only with s3v4 signature on STREAMING-AWS4-HMAC-SHA256-PAYLOAD transfer.

Debug log

2023-03-03T14:29:28.717+0000 7fb6635d6700 1 beast: 0x7fb80c4846e0: 10.21.1.11 - 30200495337a417aa322288e36a67d9d [03/Mar/2023:14:29:28.713 +0000] "GET /warp-benchmark-bucket/?delimiter=&encoding-type=url
&fetch-owner=true&list-type=2&prefix= HTTP/1.1" 200 295 - "MinIO (linux; amd64) minio-go/v7.0.47 warp/0.6.6" - latency=0.004000011s
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 CONTENT_LENGTH=17934
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 CONTENT_TYPE=application/octet-stream
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-
date;x-amz-decoded-content-length,Signature=15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_CONNECTION=close
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_HOST=object.az1.devtest.local
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_USER_AGENT=MinIO (linux; amd64) minio-go/v7.0.47 warp/0.6.6
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_VERSION=1.1
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_CONTENT_SHA256=STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_DATE=20230303T142928Z
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_AMZ_DECODED_CONTENT_LENGTH=17759
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 HTTP_X_FORWARDED_FOR=10.21.1.11
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REMOTE_ADDR=10.21.1.11
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REQUEST_METHOD=PUT
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 REQUEST_URI=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 SCRIPT_URI=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 SERVER_PORT=8080
2023-03-03T14:29:28.721+0000 7fb716f3d700 1 ====== starting new request req=0x7fb80c4846e0 =====
2023-03-03T14:29:28.721+0000 7fb716f3d700 2 req 13682662579721275793 0.000000000s initializing for trans_id = tx00000bde2951f55d10991-0064020448-2d61e9-az1
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s rgw api priority: s3=8 s3website=7
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s host=object.az1.devtest.local
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_ur
i=/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_CONTENT_SHA256
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_DATE
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s meta>> HTTP_X_AMZ_DECODED_CONTENT_LENGTH
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-date:20230303T142928Z
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s x>> x-amz-decoded-content-length:17759
2023-03-03T14:29:28.721+0000 7fb716f3d700 20 req 13682662579721275793 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3
2023-03-03T14:29:28.721+0000 7fb716f3d700 10 req 13682662579721275793 0.000000000s handler=22RGWHandler_REST_Obj_S3
2023-03-03T14:29:28.721+0000 7fb716f3d700 2 req 13682662579721275793 0.000000000s getting op 1
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj scheduling with throttler client=2 cost=1
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj op=21RGWPutObj_ObjStore_S3
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj verifying requester
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s payload request hash = STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request = PUT
/warp-benchmark-bucket/od5KYq9I/1.jyjnL1J%29c4qW97kS.rnd

host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s date_k = 09612b40b246d9e3ad696eb2db88b68fb33bb1df778fa95a169f3c7e497f7080
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s region_k = 42ab646d3cc2db152885e329f805189d27eef5974f92f5d832f69b598ae421a3
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s service_k = aa55192295115a5c79dc3d5f0bed60f6a39dfdc3bfa37b469b7b0e35e708f3b6
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s signing_k = 3446bc83dab69902f2f197250c08f4b849a5d48e265c62b55295dff3ae9601d6
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s generated signature = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj s3 keystone: validated token: stor_test:stor_test expires: 1677896734
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::keystone::EC2Engine denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj get_system_obj_state: rctx=0x7fb80c482c68 obj=az1.rgw.meta:users.keys:798869c7266e4965aa5151f9a1924083 state=0
x7fb800253d00 s->prefetch_data=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj cache get: name=az1.rgw.meta+users.keys+798869c7266e4965aa5151f9a1924083 : hit (negative entry)
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj error reading user info, uid=798869c7266e4965aa5151f9a1924083 can't authenticate
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::LocalEngine denied with reason=-2028
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 signature format = 15fd42a5005386bd42e2847a4efdf178ca905627c1e0414c87d779048fd14958
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s v4 credential format = 798869c7266e4965aa5151f9a1924083/20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s access key id = 798869c7266e4965aa5151f9a1924083
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s credential scope = 20230303/devtest/s3/aws4_request
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical headers format = host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

host:object.az1.devtest.local
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230303T142928Z
x-amz-decoded-content-length:17759

host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s canonical request hash = 09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s string to sign = AWS4-HMAC-SHA256
20230303T142928Z
20230303/devtest/s3/aws4_request
09827dcad8a640059efb5d7ebb6ace24e98a0ced7d2fbca027afb418b95066da
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s body content detected in multiple chunks
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s aws4 seed signature ok... delaying v4 auth
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj get_system_obj_state: rctx=0x7fb80c482c68 obj=az1.rgw.meta:users.keys:798869c7266e4965aa5151f9a1924083 state=0
x7fb800253d00 s->prefetch_data=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 req 13682662579721275793 0.004000011s s3:put_obj cache get: name=az1.rgw.meta+users.keys+798869c7266e4965aa5151f9a1924083 : hit (negative entry)
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj error reading user info, uid=798869c7266e4965aa5151f9a1924083 can't authenticate
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::LocalEngine denied with reason=-2028
2023-03-03T14:29:28.725+0000 7fb716f3d700 20 req 13682662579721275793 0.004000011s s3:put_obj rgw::auth::s3::AWSAuthStrategy denied with reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 5 req 13682662579721275793 0.004000011s s3:put_obj Failed the auth strategy, reason=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 10 failed to authorize request
2023-03-03T14:29:28.725+0000 7fb716f3d700 1 req 13682662579721275793 0.004000011s op->ERRORHANDLER: err_no=-2201 new_err_no=-2201
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj op status=0
2023-03-03T14:29:28.725+0000 7fb716f3d700 2 req 13682662579721275793 0.004000011s s3:put_obj http status=501
2023-03-03T14:29:28.725+0000 7fb716f3d700 1 ====== req done req=0x7fb80c4846e0 op status=0 http_status=501 latency=0.004000011s ======

(in above real hostname and region name were obfuscated)

The correct behaviour is to accept PUT and continue upload. This works with a local rgw user or with signature s3v2. Other operations/methods like GET/STAT/DELETE works fine with s3v4 and Keystone.

Related issues 3 (1 open — 2 closed)

Actions

Copy link

Updated by Pawel Stefanski about 1 year ago

Of course please set target version accordingly to release plans, redmine works so badly I've missed it's set.

Actions

Copy link

Updated by Casey Bodley about 1 year ago

the NotImplemented error is coming from https://github.com/ceph/ceph/blob/950a61a/src/rgw/rgw_auth_s3.cc#L1232-L1243, where a comment says:

    /* Some external authorizers (like Keystone) aren't fully compliant with
     * AWSv4. They do not provide the secret_key which is necessary to handle
     * the streamed upload. */

i'm not familiar with the details here

Actions

Copy link

Updated by Casey Bodley about 1 year ago

Status changed from New to Triaged

Actions

Copy link

Updated by Casey Bodley about 1 year ago

Subject changed from Object upload doesn't work with chunked upload and Keystone auth. to Keystone EC2 auth does not support STREAMING-AWS4-HMAC-SHA256-PAYLOAD

Actions

Copy link

Updated by Pawel Stefanski about 1 year ago

Thanks Casey for evaluation, yes, I found similar reports for LDAP ext auth as well, so there is definitely interest in the community for this feature.
Is there any ongoing effort to support it ?

Actions

Copy link

Updated by Casey Bodley about 1 year ago

Status changed from Triaged to Fix Under Review
Backport set to pacific quincy reef
Pull request ID set to 50550

hey Pawel, i spent some more time looking into this, and reviewed the last EC2 changes from https://github.com/ceph/ceph/pull/26095

it turns out that rgw does have access to the secret key needed to implement STREAMING-AWS4-HMAC-SHA256-PAYLOAD, it just wasn't being passed into that function that was returning NOT_IMPLEMENTED

i've raised a PR, but afaik we don't have any test coverage for EC2. if i were to produce a build or container with this change, would you be willing to help with testing?

Actions

Copy link

Updated by Pawel Stefanski about 1 year ago

yes, sure, awesome news. I will be more than happy to test it with a Keystone instance. Deb builds would be awesome, containers but will take a bit more time.

Actions

Copy link

Updated by Casey Bodley about 1 year ago

Pawel Stefanski wrote:

yes, sure, awesome news. I will be more than happy to test it with a Keystone instance. Deb builds would be awesome, containers but will take a bit more time.

thanks Pawel; assuming it's most convenient for you to test the fix against the pacific release, i've scheduled builds at https://shaman.ceph.com/builds/ceph/wip-58908-pacific/. once those finish, the rpm and deb repos should be available under https://shaman.ceph.com/repos/ceph/wip-58908-pacific/

if you prefer testing against main, the same builds are scheduled under the branch 'wip-58908'

Actions

Copy link

Updated by Casey Bodley about 1 year ago

Assignee set to Casey Bodley

Actions

Copy link

#10

Updated by Pawel Stefanski about 1 year ago

I've just tested it with Keystone at Xena version,

it works, the same test is passing clean.
I've testes your build (ceph version 16.2.11-305-gc4c69ea1 (c4c69ea14de4fad0791b62dc7dbd1357ebf8bdff) pacific (stable))

2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s initializing for trans_id = tx00000a7f03d4d7ad31a5a-00641c2c08-3794-default
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s rgw api priority: s3=8 s3website=7
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s host=localhost
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s meta>> HTTP_X_AMZ_CONTENT_SHA256
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s meta>> HTTP_X_AMZ_DATE
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s meta>> HTTP_X_AMZ_DECODED_CONTENT_LENGTH
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s x>> x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s x>> x-amz-date:20230323T103800Z
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s x>> x-amz-decoded-content-length:16833
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s handler=22RGWHandler_REST_Obj_S3
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s getting op 1
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj scheduling with throttler client=2 cost=1
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj op=21RGWPutObj_ObjStore_S3
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj verifying requester
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s v4 signature format = d4b3b5014411743b2986e5969e31cf514cf7c7840792e5cf1aa03848a50dd6e
4
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s v4 credential format = 074a9a1fd111405f9b62650ed9c26a3a/20230323/default/s3/aws4_requ
est
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s access key id = 074a9a1fd111405f9b62650ed9c26a3a
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s credential scope = 20230323/default/s3/aws4_request
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s canonical headers format = host:localhost:8080
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230323T103800Z
x-amz-decoded-content-length:16833
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s payload request hash = STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s canonical request = PUT
/warp-benchmark-bucket/rqV9myKH/68.wrYt9sM4Es%29oPKrm.rnd

host:localhost:8080
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20230323T103800Z
x-amz-decoded-content-length:16833

host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s canonical request hash = dcb411a42cc82c2dee0025a4b0f9563a3168d619a3671456e367d8995b53886b
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s canonical request hash = dcb411a42cc82c2dee0025a4b0f9563a3168d619a3671456e367d8995b53886b
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s string to sign = AWS4-HMAC-SHA256
20230323T103800Z
20230323/default/s3/aws4_request
dcb411a42cc82c2dee0025a4b0f9563a3168d619a3671456e367d8995b53886b
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s body content detected in multiple chunks
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s aws4 seed signature ok... delaying v4 auth
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s date_k    = e928be3c7486520cbcfc5793e73548cdc7a5ffff2cf893f4d606f855742cddd9
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s region_k  = 1cccfe75b8428d591e9d64cad2f2280f0e96ac7d620fa036a2b1cfa74c2941ea
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s service_k = f531bd76ed947ab2131172b8c36f68a9473ed1b7d50a090d56a905755bbec7dc
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s signing_k = 1f7bb38710d70403a3119c2c856a983f9a65ce8ba965885ceb9a5297eec1346c
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s generated signature = d4b3b5014411743b2986e5969e31cf514cf7c7840792e5cf1aa03848a50dd6e4
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj s3 keystone: validated token: demo:admin expires: 1679571346
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s date_k    = e928be3c7486520cbcfc5793e73548cdc7a5ffff2cf893f4d606f855742cddd9
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s region_k  = 1cccfe75b8428d591e9d64cad2f2280f0e96ac7d620fa036a2b1cfa74c2941ea
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s service_k = f531bd76ed947ab2131172b8c36f68a9473ed1b7d50a090d56a905755bbec7dc
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s signing_k = 1f7bb38710d70403a3119c2c856a983f9a65ce8ba965885ceb9a5297eec1346c
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+users.uid+28f284c52f44409d9faa5480851e1666$28f284c52f44409d9faa5480851e1666 : hit (negative entry)
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+users.uid+28f284c52f44409d9faa5480851e1666 : hit (requested=0x6, cached=0x7)
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+users.uid+28f284c52f44409d9faa5480851e1666 : hit (requested=0x1, cached=0x7)
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj normalizing buckets and tenants
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s->object=rqV9myKH/68.wrYt9sM4Es)oPKrm.rnd s->bucket=warp-benchmark-bucket
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj init permissions
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+root+warp-benchmark-bucket : hit (requested=0x16, cached=0x17)
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+root+warp-benchmark-bucket : hit (requested=0x11, cached=0x17)
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+users.uid+28f284c52f44409d9faa5480851e1666 : hit (requested=0x6, cached=0x7)
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj cache get: name=default.rgw.meta+users.uid+28f284c52f44409d9faa5480851e1666 : hit (requested=0x3, cached=0x7)
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj recalculating target
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj reading permissions
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj init op
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj verifying op mask
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj verifying op permissions
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=28f284c52f44409d9faa5480851e1666, acct_name=demo, perm_mask=15, is_admin=0) mask=50
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Searching permissions for uid=28f284c52f44409d9faa5480851e166
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Found permission: 15
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Searching permissions for uid=28f284c52f44409d9faa5480851e1666$28f284c52f44409d9faa5480851e1666
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Permissions for user not found
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Searching permissions for group=1 mask=50
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Permissions for group not found
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Searching permissions for group=2 mask=50
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj Permissions for group not found
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=28f284c52f44409d9faa5480851e1666, acct_name=demo, perm_mask=15, is_admin=0), owner=28f284c52f44409d9faa5480851e1666, perm=2
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj  identity=rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=28f284c52f44409d9faa5480851e1666, acct_name=demo, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj verifying op params
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj pre-executing
2023-03-23T10:38:00.535+0000 7fdae3faf700  2 req 12101239601726954074 0.000000000s s3:put_obj executing
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj NOTICE: call to do_aws4_auth_completion
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj v4 auth ok -- do_aws4_auth_completion
2023-03-23T10:38:00.535+0000 7fdae3faf700  5 req 12101239601726954074 0.000000000s s3:put_obj NOTICE: call to do_aws4_auth_completion
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj x>> x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj x>> x-amz-date:20230323T103800Z
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj x>> x-amz-decoded-content-length:16833
2023-03-23T10:38:00.535+0000 7fdae3faf700 10 req 12101239601726954074 0.000000000s s3:put_obj setting object write_tag=3eeb8e6f-0f8d-48c6-b736-60993abd96eb.14228.12101239601726954074
2023-03-23T10:38:00.543+0000 7fda796da700 10 req 12101239601726954074 0.008000574s s3:put_obj cache get: name=default.rgw.log++bucket.sync-source-hints.warp-benchmark-bucket : hit (negative entry)
2023-03-23T10:38:00.543+0000 7fda796da700 10 req 12101239601726954074 0.008000574s s3:put_obj cache get: name=default.rgw.log++bucket.sync-target-hints.warp-benchmark-bucket : hit (negative entry)
2023-03-23T10:38:00.543+0000 7fda796da700 10 req 12101239601726954074 0.008000574s s3:put_obj chain_cache_entry: cache_locator=
2023-03-23T10:38:00.543+0000 7fda796da700  2 req 12101239601726954074 0.008000574s s3:put_obj completing
2023-03-23T10:38:00.543+0000 7fda796da700  2 req 12101239601726954074 0.008000574s s3:put_obj op status=0
2023-03-23T10:38:00.543+0000 7fda796da700  2 req 12101239601726954074 0.008000574s s3:put_obj http status=200
2023-03-23T10:38:00.543+0000 7fda796da700  1 ====== req done req=0x7fdb606735e0 op status=0 http_status=200 latency=0.008000574s ======

thank you soo much!!!!

Actions

Copy link

#11