Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2024-02-12T07:32:26ZCeph
Redmine rgw - Cleanup #64388 (New): improve error message when expired pre-signed urls are being usedhttps://tracker.ceph.com/issues/643882024-02-12T07:32:26ZPritha Srivastavaprsrivas@redhat.com
<p>improve the error message that is logged and that is returned back to the client, in case an expired pre-signed URL is used.</p> rgw - Feature #63214 (New): rgw: update thumbprint list of an oidc providerhttps://tracker.ceph.com/issues/632142023-10-16T07:51:02ZPritha Srivastavaprsrivas@redhat.com
<p>the implementation of oidc provider in rgw currently doesnt support updating thumbprint list of an existing oidc provider.<br />this tracker tracks implementation of the same.</p> rgw - Feature #63213 (New): rgw:add client id to open id connect providerhttps://tracker.ceph.com/issues/632132023-10-16T07:49:31ZPritha Srivastavaprsrivas@redhat.com
<p>The implementation of openid connect provider in rgw currently does not provide the feature to add client id to an existing openid connect provider.<br />This tracker tracks the implementation of the same.</p> rgw - Bug #62541 (Triaged): docs: sts-AssumeRoleWithWebIdentity does not work for tenanted roleshttps://tracker.ceph.com/issues/625412023-08-23T07:09:42ZGuenter Sandner
<p>having two roles using the same assume-role-policy-document defined like this<br /><pre>
# create non-tenanted role
radosgw-admin role create --role-name='devS3Access' --path=/ \
--assume-role-policy-doc='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/192.168.100.69:9080/realms/aurora-dev"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"192.168.100.69:9080/realms/aurora-dev:azp":"aurora-api-gateway-dev"}}}]}'
# create tenanted role
radosgw-admin role create --tenant tenant1 --role-name='tenant1S3Access' --path=/ \
--assume-role-policy-doc='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/192.168.100.69:9080/realms/aurora-dev"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"192.168.100.69:9080/realms/aurora-dev:azp":"aurora-api-gateway-dev"}}}]}'
</pre><br />calling "sts asume-role-with-web-identity" works for the non tenanted role "devS3Access", but not for the tenanted role "tenant1S3Access"</p>
<pre>
[root@rook-ceph-tools-855599bf84-wpwhr /]# aws --endpoint=http://$AWS_HOST:$PORT sts assume-role-with-web-identity \
> --role-arn 'arn:aws:iam::tenant1:role/S3Access' \
> --role-session-name 'tenant1S3Access' \
> --web-identity-token "${ID_TOKEN_DEV}" \
> --duration-seconds 3600 >tenant1_secrets || echo FAILED
An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown
</pre>
<p>in the logs we can see this for the non-tenanted role:<br /><pre>
debug 2023-08-22T11:28:14.541+0000 7f2b2359c700 10 req 18142818711300298359 0.000000000s sts:assume_role_web_identity cache get: name=my-store.rgw.meta+oidc+oidc_url.192.168.100.69:9080/realms/aurora-dev : hit (requested=0x1, cached=0x7)
</pre><br />but this for the tenanted role:<br /><pre>
ebug 2023-08-22T11:23:44.114+0000 7f2b78646700 10 req 13090842789017334974 0.001000009s sts:assume_role_web_identity cache get: name=my-store.rgw.meta+oidc+tenant1oidc_url.192.168.100.69:9080/realms/aurora-dev : hit (negative entry)
debug 2023-08-22T11:23:44.114+0000 7f2b78646700 0 req 13090842789017334974 0.001000009s sts:assume_role_web_identity Couldn't get oidc provider info using input isshttp://192.168.100.69:9080/realms/aurora-dev
</pre></p> rgw - Bug #61916 (Pending Backport): qa/sts: test_list_buckets_invalid_auth and test_list_buckets...https://tracker.ceph.com/issues/619162023-07-06T13:54:16ZCasey Bodleycbodley@redhat.com
<p>until <a class="external" href="https://github.com/ceph/ceph/pull/52216">https://github.com/ceph/ceph/pull/52216</a>, we <strong>only</strong> ran the test cases marked <code>sts_tests</code> and <code>webidentity_tests</code> in this configuration. now that we run all test cases, these two ListBuckets tests are failing with the wrong error code</p>
<p>ex. <a class="external" href="http://qa-proxy.ceph.com/teuthology/cbodley-2023-06-30_13:14:21-rgw-main-distro-default-smithi/7322679/teuthology.log">http://qa-proxy.ceph.com/teuthology/cbodley-2023-06-30_13:14:21-rgw-main-distro-default-smithi/7322679/teuthology.log</a></p> rgw - Backport #61633 (New): quincy: failure of cls_rgw_gc_list is not handledhttps://tracker.ceph.com/issues/616332023-06-09T17:26:46ZBackport Botrgw - Backport #61631 (New): reef: failure of cls_rgw_gc_list is not handledhttps://tracker.ceph.com/issues/616312023-06-09T17:26:31ZBackport Botrgw - Backport #59607 (New): reef: Renaming large files fails with 403 when using temporary creds...https://tracker.ceph.com/issues/596072023-05-02T20:59:46ZBackport Botrgw - Backport #59605 (New): quincy: Renaming large files fails with 403 when using temporary cre...https://tracker.ceph.com/issues/596052023-05-02T20:59:32ZBackport Botrgw - Backport #59274 (New): quincy: STS AssumeRoleWithWebIdentity improper url concatenation of ...https://tracker.ceph.com/issues/592742023-03-31T17:51:09ZBackport Botrgw - Bug #58890 (Pending Backport): STS AssumeRoleWithWebIdentity improper url concatenation of ...https://tracker.ceph.com/issues/588902023-03-01T14:46:38ZMathew Uttermat@hazmat.dev
<p>When attempting to utilize the <code>AssumeRoleWithWebIdentity</code> STS API <code>rgw_rest_sts.cc#L312</code> will attempt a HTTP GET to an improperly concatenated URL if the ISS from the JWT token ends in a slash (<code>/</code>). The result is a URL like <code>https://myprovider//.well-known/openid-configuration</code> which is not a well-formed URL and results in undefined behavior with different identity providers (case 1). There is also an alternative behavior when the configured OpenIDConnect provider does not contain the trailing <code>/</code> and the ISS does (case 2).</p>
<p>Keycloak - This is fine and the ISS does not contain a ending slash.<br />Authentik - The server will respond with a 301 redirect to the valid URL (without <code>//</code>) and RGWHTTPTransceiver does not follow redirects so request fails.</p>
<p>Source: <a class="external" href="https://github.com/ceph/ceph/blob/cca84e653dd5ea686884cb85fdd8e20703678274/src/rgw/rgw_rest_sts.cc#L312">https://github.com/ceph/ceph/blob/cca84e653dd5ea686884cb85fdd8e20703678274/src/rgw/rgw_rest_sts.cc#L312</a></p>
<p><strong>Suggested Fix</strong></p>
<p>Trim the trailing <code>/</code> from the ISS before concatenating the <code>/.well-known/openid-configuration</code> path suffix.</p>
<p><strong>Logs</strong></p>
<p>These are logs from the AssumeRoleWithWebIdentity API call. My logs are slightly altered for readability. This is also an offline test lab so I do not care that I am leaking credentials/identity.</p>
<p>(case 1) OpenIDConnect provider is configured with a URL of <code>https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/</code> and matches the ISS.</p>
<pre>
1 ====== starting new request req=0x7efc3c0c8680 =====
initializing for trans_id = tx000002dda8f6818905e64-0063fec66d-3607b8-default
rgw api priority: s3=5 s3website=4
host=s3.lab
subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0
final domain/bucket subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.lab s->info.request_uri=/
get_handler handler=26RGWHandler_REST_Service_S3
handler=26RGWHandler_REST_Service_S3
getting op 4
Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjI5MDc3ODY2YmNiMGQ3NWI3ZDJlNTFmZTQ1NDA1Yzk3IiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2xvZ2luLmxhYi9hcHBsaWNhdGlvbi9vL2Q3ZDY0NDk2ZTI2YzE1NmNhOWVhMDgwMmM1ZDdlZDFjLyIsInN1YiI6Im1hdGhldy51dHRlciIsImF1ZCI6InJhZG9zZ3ciLCJleHAiOjE2Nzc2NDMxMjMsImlhdCI6MTY3NzY0MTMyMywiYXV0aF90aW1lIjoxNjc3NjQxMzIzLCJhY3IiOiJnb2F1dGhlbnRpay5pby9wcm92aWRlcnMvb2F1dGgyL2RlZmF1bHQiLCJlbWFpbCI6Im1hdGhldy51dHRlckBzaGlmdDUuaW8iLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6Im1hdGhldy51dHRlciIsImdpdmVuX25hbWUiOiJtYXRoZXcudXR0ZXIiLCJmYW1pbHlfbmFtZSI6IiIsInByZWZlcnJlZF91c2VybmFtZSI6Im1hdGhldy51dHRlciIsIm5pY2tuYW1lIjoibWF0aGV3LnV0dGVyIiwiZ3JvdXBzIjpbImxhYnMtdXNlciIsInB2ZS11c2VyIiwiYXV0aGVudGlrIEFkbWlucyIsInMzLXVzZXIiXSwiY2lkIjoicmFkb3NndyIsInVpZCI6IlBvXn1FfGMzKmhxTHsxfU8_YUtzIS1nRTpDVzslMHYtcUlsZy01W2RwOXspb3VIcyxSOGE3bmt-ell2RkU8WzZCIUduaC4yMFQyQkBNJy10UylzQ2Q2dnJmJTtoLXwxMW95ZV4vLnVFTXo1NzJsRTcuSDBpdDBeJz58LWNNenopIn0.ZphoRcXj7gDfJl1zNhk5GBZvjcQlbWpcUp5GLOq5jJ50KVQG1c0zaKES7uqjC-AlNRS_r-Mp1x2a2kCqT_F_MgFKalfuajUif23g-kS8eVDEm16pdzW2_O_tKaOcI_BlBqu0izb-pXGc9TLrCclZ2n87ZGQZxKaJx2maGk7lpIV-IQ__b2ftxO3AewL1RSxOneAakXtCgZSd5ye-UgG5WKzBaIySeCvp1TaezZW_zvmMy-Z4EHMwkhkdzhOUJpvkFL_0FHsTQUV4Ws37CHinCvx6IbC9Mj2gBGf-aXbxT1aMPEuC9JsNmPavHKg9nqvwJ_RC7FycdJANQY0tMGrArppReSZprEywxuJmz1vUqNCdhPWrqLSoAxoOiaYPTVZPcuXs62TH5jeI9ysp2YVo_5iu0Tfsl7f27PxzdlH_PfgWzJt5Yit1T9xYPTbG98FlV3D8x4NJ1AF-_KGtqE8FbvKWsqs05qd24jyYEoutAiEReDySFrXal8uctGsVcmyxEg0cr7R9GsAMi4J3aDrYK7eNWWcjjlICM5qWfUHvkba-wtkHrIR-Rd_UmdyV1t9iixQzagfKjCQxRxpJKPtTz_OunaLeNZffE04xF2PxrFMGX0PoCSICE-hW4ODtDKa5Yq_omTLWNTGHc65mMQkIlmuD_R556mBbRbR7McamF98&RoleSessionName=mathew.utter&ProviderId=login.lab&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FAssumeRoleWithWebIdentityForOIDC
get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.prerequest. state=0x7ef734005260 s->prefetch_data=0
cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)
sts:assume_role_web_identity scheduling with throttler client=0 cost=1
sts:assume_role_web_identity op=31RGWSTSAssumeRoleWithWebIdentity
sts:assume_role_web_identity verifying requester
sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine
sts:assume_role_web_identity payload = {"iss":"https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/","sub":"mathew.utter","aud":"radosgw","exp":1677643123,"iat":1677641323,"auth_time":1677641323,"acr":"goauthentik.io/providers/oauth2/default","email":"mathew.utter@shift5.io","email_verified":true,"name":"mathew.utter","given_name":"mathew.utter","family_name":"","preferred_username":"mathew.utter","nickname":"mathew.utter","groups":["labs-user","pve-user","authentik Admins","s3-user"],"cid":"radosgw","uid":"Po^}E|c3*hqL{1}O?aKs!-gE:CW;%0v-qIlg-5[dp9{)ouHs,R8a7nk~zYvFE<[6B!Gnh.20T2B@M'-tS)sCd6vrf%;h-|11oye^/.uEMz572lE7.H0it0^'>|-cMzz)"}
sts:assume_role_web_identity get_system_obj_state: rctx=0x7efc3c0c6b70 obj=default.rgw.meta:oidc:oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ state=0x7ef734005260 s->prefetch_data=0
sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : hit (requested=0x6, cached=0x7)
sts:assume_role_web_identity get_system_obj_state: s->obj_tag was set empty
sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : hit (requested=0x1, cached=0x7)
20 sending request to https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c//.well-known/openid-configuration
20 register_request mgr=0x559fd8a118f0 req_data->id=4, curl_handle=0x7ef73400bcd0
20 link_request req_data=0x7ef734010550 req_data->id=4, curl_handle=0x7ef73400bcd0
sts:assume_role_web_identity HTTP request res: -5
sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13
sts:assume_role_web_identity Failed the auth strategy, reason=-13
10 failed to authorize request
op->ERRORHANDLER: err_no=-13 new_err_no=-13
get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.postrequest. state=0x7ef790000f70 s->prefetch_data=0
cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
sts:assume_role_web_identity op status=0
sts:assume_role_web_identity http status=403
1 ====== req done req=0x7efc3c0c8680 op status=0 http_status=403 latency=0.028000288s ======
</pre>
<p>(case 2) OpenIDConnect provider is configured with a URL of <code>https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c</code> and does not match the ISS because of missing <code>/</code>.</p>
<pre>
1 ====== starting new request req=0x7efc3c0c8680 =====
initializing for trans_id = tx00000a51169179fd21dcc-0063fecf2a-3607b8-default
rgw api priority: s3=5 s3website=4
host=s3.lab
subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0
final domain/bucket subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.lab s->info.request_uri=/
get_handler handler=26RGWHandler_REST_Service_S3
handler=26RGWHandler_REST_Service_S3
getting op 4
Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjI5MDc3ODY2YmNiMGQ3NWI3ZDJlNTFmZTQ1NDA1Yzk3IiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2xvZ2luLmxhYi9hcHBsaWNhdGlvbi9vL2Q3ZDY0NDk2ZTI2YzE1NmNhOWVhMDgwMmM1ZDdlZDFjLyIsInN1YiI6Im1hdGhldy51dHRlciIsImF1ZCI6InJhZG9zZ3ciLCJleHAiOjE2Nzc2NDUzNjAsImlhdCI6MTY3NzY0MzU2MCwiYXV0aF90aW1lIjoxNjc3NjQzNTYwLCJhY3IiOiJnb2F1dGhlbnRpay5pby9wcm92aWRlcnMvb2F1dGgyL2RlZmF1bHQiLCJlbWFpbCI6Im1hdGhldy51dHRlckBzaGlmdDUuaW8iLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6Im1hdGhldy51dHRlciIsImdpdmVuX25hbWUiOiJtYXRoZXcudXR0ZXIiLCJmYW1pbHlfbmFtZSI6IiIsInByZWZlcnJlZF91c2VybmFtZSI6Im1hdGhldy51dHRlciIsIm5pY2tuYW1lIjoibWF0aGV3LnV0dGVyIiwiZ3JvdXBzIjpbImxhYnMtdXNlciIsInB2ZS11c2VyIiwiYXV0aGVudGlrIEFkbWlucyIsInMzLXVzZXIiXSwiY2lkIjoicmFkb3NndyIsInVpZCI6Incja1t-Mm9bbDRKJF9TK3prPWxWLXw6b2A8P05yaVdvbWlcIiRiL2ItZi5ocCZGQT86UmM0R3p6K04zQSxEXCJ6VU5gfTxDbTkobjJhLkBeKTV7bk9-MTx3Wzpze1ZEPjo3UUduLjtyP3R5Jys7WFxcIX5TYFY_e2s5Myg7b3g3e3MqIn0.iPO-aTBtkTXwI8QDGWL0IbRwSAOwAVcofctVZpGATtQer59K8gCE0PlzBj_mMyd1Vge__W5HCrorfQNkDnI5ekvToc3tjsptY0gTAAFEfTFrmK7thLoxAsawRzcxxsmLFjHO8E0i2it-OLITMVucVzi5kKobUs5uR7TFZLHe39yslI2Ux3z6iBMe7Pb6eSZh36xiQZ7-mHFSZu05Zt6j8rg8yB9k0ckZZg8uQwhp8-E5KHdmkzUaWpldCHI73XiYH7gZVT3mJgFvAhLMFvhr96kgOT0cUKuNx3iQBChV7c-1_mlcDYbQkuZfvzlSqGAa0tdBMSX13Q9cOgw4i0S9i7ApGwyY5C5PXaqOTIgkEB91hcziUZuiWisT5BLFbgb-Mv4OmU1iA4w26a9Jl4bdtY_KPJwMkHZfOW_WqB27vZSG_DwInrqSXaMBr-mUU3sPralrIBF75WhmcY8iNRik136oXUka3WiJLcG4hTIT8AwSziISTdzyqS9nDL9OJkCoivZrjZuFLffbhHV2NXJt3bUQjEZqzpkznmyVvNYabubqm-rG0-Nu8czuf5MlxdmiqDCB0JnpVJE0XeGie0hWg1TJbrJe0N3z3EYs__82mSyMg5ifG_H0QTWzOvx2SKfNLE74kpgUYlwTWBth-kZPp0rUm7spiBurKXD3AKpXWx4&RoleSessionName=mathew.utter&ProviderId=login.lab&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FAssumeRoleWithWebIdentityForOIDC
get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.prerequest. state=0x7ef618004500 s->prefetch_data=0
cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)
sts:assume_role_web_identity scheduling with throttler client=0 cost=1
sts:assume_role_web_identity op=31RGWSTSAssumeRoleWithWebIdentity
sts:assume_role_web_identity verifying requester
sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine
sts:assume_role_web_identity payload = {"iss":"https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/","sub":"mathew.utter","aud":"radosgw","exp":1677645360,"iat":1677643560,"auth_time":1677643560,"acr":"goauthentik.io/providers/oauth2/default","email":"mathew.utter@shift5.io","email_verified":true,"name":"mathew.utter","given_name":"mathew.utter","family_name":"","preferred_username":"mathew.utter","nickname":"mathew.utter","groups":["labs-user","pve-user","authentik Admins","s3-user"],"cid":"radosgw","uid":"w#k[~2o[l4J$_S+zk=lV-|:o`<?NriWomi\"$b/b-f.hp&FA?:Rc4Gzz+N3A,D\"zUN`}<Cm9(n2a.@^)5{nO~1<w[:s{VD>:7QGn.;r?ty'+;X\\!~S`V?{k93(;ox7{s*"}
sts:assume_role_web_identity get_system_obj_state: rctx=0x7efc3c0c6b70 obj=default.rgw.meta:oidc:oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ state=0x7ef618004500 s->prefetch_data=0
sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : miss
sts:assume_role_web_identity WARNING: blocking librados call
1 -- 172.25.1.102:0/636600742 --> [v2:172.25.1.102:6800/1514588,v1:172.25.1.102:6801/1514588] -- osd_op(unknown.0.0:9038 6.a 6:512c5d0d:oidc::oidc_url.login.lab%2fapplication%2fo%2fd7d64496e26c156ca9ea0802c5d7ed1c%2f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e1115) v8 -- 0x7ef61800d9c0 con 0x7efc2c02f8a0
1 -- 172.25.1.102:0/636600742 <== osd.13 v2:172.25.1.102:6800/1514588 1915 ==== osd_op_reply(9038 oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 252+0+0 (crc 0 0 0) 0x7efc2c105040 con 0x7efc2c02f8a0
sts:assume_role_web_identity cache put: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ info.flags=0x0
sts:assume_role_web_identity adding default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ to cache LRU end
sts:assume_role_web_identity Couldn't get oidc provider info using input isshttps://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/
sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13
sts:assume_role_web_identity Failed the auth strategy, reason=-13
10 failed to authorize request
op->ERRORHANDLER: err_no=-13 new_err_no=-13
get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.postrequest. state=0x7ef618004500 s->prefetch_data=0
cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
sts:assume_role_web_identity op status=0
sts:assume_role_web_identity http status=403
1 ====== req done req=0x7efc3c0c8680 op status=0 http_status=403 latency=0.000000000s ======
</pre>
<p>No PR at the moment.</p> rgw - Bug #58628 (Pending Backport): Renaming large files fails with 403 when using temporary cre...https://tracker.ceph.com/issues/586282023-02-02T04:20:33ZPritha Srivastavaprsrivas@redhat.com
<p>Renaming large files fails with 403 when using temporary creds returned by STS. These files are uploaded using MultipartUpload. aws s3 cli uses 8MB as the limit and s3cmd uses 5MB.</p> rgw - Bug #58547 (Need More Info): ?Action=ListRoles endpoint json response incorrectly formattedhttps://tracker.ceph.com/issues/585472023-01-23T09:54:11ZPere Díaz Bou
<p>After creating 2 dummy roles and trying to retrieve the list of roles through the S3 api, the list of roles was inadequately formatted. The roles were created following the docs:</p>
<pre><code class="c syntaxhl"><span class="CodeRay">radosgw-admin role create --role-name=S3Access1 --path=/application_abc/component_xyz/ --assume-role-policy-doc=<span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Version</span><span class="char">\"</span><span class="content">:</span><span class="char">\"</span><span class="content">2012-10-17</span><span class="char">\"</span><span class="content">,</span><span class="char">\"</span><span class="content">Statement</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Effect</span><span class="char">\"</span><span class="content">:</span><span class="char">\"</span><span class="content">Allow</span><span class="char">\"</span><span class="content">,</span><span class="char">\"</span><span class="content">Principal</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">AWS</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">arn:aws:iam:::user/TESTER</span><span class="char">\"</span></span><span class="error">\</span>]<span class="error">\</span>},<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Action</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">sts:AssumeRole</span><span class="char">\"</span></span><span class="error">\</span>]<span class="error">\</span>}<span class="error">\</span>]<span class="error">\</span>}
radosgw-admin role create --role-name=S3Access2 --path=/application_abc/component_xyz/ --assume-role-policy-doc=<span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Version</span><span class="char">\"</span><span class="content">:</span><span class="char">\"</span><span class="content">2012-10-17</span><span class="char">\"</span><span class="content">,</span><span class="char">\"</span><span class="content">Statement</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Effect</span><span class="char">\"</span><span class="content">:</span><span class="char">\"</span><span class="content">Allow</span><span class="char">\"</span><span class="content">,</span><span class="char">\"</span><span class="content">Principal</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>{<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">AWS</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">arn:aws:iam:::user/TESTER</span><span class="char">\"</span></span><span class="error">\</span>]<span class="error">\</span>},<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">Action</span><span class="char">\"</span><span class="content">:</span></span><span class="error">\</span>[<span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">sts:AssumeRole</span><span class="char">\"</span></span><span class="error">\</span>]<span class="error">\</span>}<span class="error">\</span>]<span class="error">\</span>}
</span></code></pre>
<p>After calling the endpoint the response received was unexpectedly transformed:<br />RAW response:<br /><pre><code class="c syntaxhl"><span class="CodeRay">b<span class="char">'[</span>[{<span class="string"><span class="delimiter">"</span><span class="content">member</span><span class="delimiter">"</span></span>:{<span class="string"><span class="delimiter">"</span><span class="content">RoleId</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">904622a9-62d2-4e3a-b996-1db6d81991a2</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">RoleName</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">S3Access2</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Path</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">/application_abc/component_xyz/</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Arn</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">arn:aws:iam:::role/application_abc/component_xyz/S3Access2</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">CreateDate</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">2023-01-23T08:46:07.624Z</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">MaxSessionDuration</span><span class="delimiter">"</span></span>:<span class="integer">3600</span>,<span class="string"><span class="delimiter">"</span><span class="content">AssumeRolePolicyDocument</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">{</span><span class="char">\\</span><span class="delimiter">"</span></span>Version<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:</span><span class="char">\\</span><span class="delimiter">"</span></span><span class="integer">2012</span>-<span class="integer">10</span>-<span class="integer">17</span><span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="char">\\</span><span class="delimiter">"</span></span>Statement<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[{</span><span class="char">\\</span><span class="delimiter">"</span></span>Effect<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:</span><span class="char">\\</span><span class="delimiter">"</span></span>Allow<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="char">\\</span><span class="delimiter">"</span></span>Principal<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:{</span><span class="char">\\</span><span class="delimiter">"</span></span>AWS<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[</span><span class="char">\\</span><span class="delimiter">"</span></span>arn:aws:iam:::user/TESTER<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">]},</span><span class="char">\\</span><span class="delimiter">"</span></span>Action<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[</span><span class="char">\\</span><span class="delimiter">"</span></span>sts:AssumeRole<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">]}]}</span><span class="delimiter">"</span></span>*},<span class="string"><span class="delimiter">"</span><span class="content">member*</span><span class="delimiter">"</span></span>:{<span class="string"><span class="delimiter">"</span><span class="content">RoleId</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">67691833-c6ab-4172-bff1-9deb176a486f</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">RoleName</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">S3Access1</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Path</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">/application_abc/component_xyz/</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Arn</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">arn:aws:iam:::role/application_abc/component_xyz/S3Access1</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">CreateDate</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">2023-01-23T08:45:49.729Z</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">MaxSessionDuration</span><span class="delimiter">"</span></span>:<span class="integer">3600</span>,<span class="string"><span class="delimiter">"</span><span class="content">AssumeRolePolicyDocument</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">{</span><span class="char">\\</span><span class="delimiter">"</span></span>Version<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:</span><span class="char">\\</span><span class="delimiter">"</span></span><span class="integer">2012</span>-<span class="integer">10</span>-<span class="integer">17</span><span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="char">\\</span><span class="delimiter">"</span></span>Statement<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[{</span><span class="char">\\</span><span class="delimiter">"</span></span>Effect<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:</span><span class="char">\\</span><span class="delimiter">"</span></span>Allow<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="char">\\</span><span class="delimiter">"</span></span>Principal<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:{</span><span class="char">\\</span><span class="delimiter">"</span></span>AWS<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[</span><span class="char">\\</span><span class="delimiter">"</span></span>arn:aws:iam:::user/TESTER<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">]},</span><span class="char">\\</span><span class="delimiter">"</span></span>Action<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">:[</span><span class="char">\\</span><span class="delimiter">"</span></span>sts:AssumeRole<span class="error">\</span><span class="error">\</span><span class="string"><span class="delimiter">"</span><span class="content">]}]}</span><span class="delimiter">"</span></span>}}],{<span class="string"><span class="delimiter">"</span><span class="content">RequestId</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">tx0000034b8fd6de729452c-0063ce4d91-1175-default</span><span class="delimiter">"</span></span>}]<span class="char">'</span>
</span></code></pre></p>
<p>After calling json.loads on the response:<br /><pre><code class="c syntaxhl"><span class="CodeRay">[[{<span class="char">'m</span>ember<span class="char">':</span> {<span class="char">'R</span>oleId<span class="char">':</span> <span class="char">'6</span><span class="integer">7691833</span>-c6ab-<span class="integer">4172</span>-bff1-<span class="integer">9</span>deb176a486f<span class="char">',</span> <span class="char">'R</span>oleName<span class="char">':</span> <span class="char">'S</span><span class="integer">3</span>Access1<span class="char">',</span> <span class="char">'P</span>ath<span class="char">':</span> <span class="char">'/</span>application_abc/component_xyz/<span class="char">',</span> <span class="char">'A</span>rn<span class="char">':</span> <span class="char">'a</span>rn:aws:iam:::role/application_abc/component_xyz/S3Access1<span class="char">',</span> <span class="char">'C</span>reateDate<span class="char">':</span> <span class="char">'2</span><span class="octal">023</span>-<span class="octal">01</span>-<span class="integer">23</span>T08:<span class="integer">45</span>:<span class="integer">4</span><span class="float">9</span><span class="float">.729</span>Z<span class="char">',</span> <span class="char">'M</span>axSessionDuration<span class="char">':</span> <span class="integer">3600</span>, <span class="char">'A</span>ssumeRolePolicyDocument<span class="char">':</span> <span class="char">'{</span><span class="string"><span class="delimiter">"</span><span class="content">Version</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">2012-10-17</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Statement</span><span class="delimiter">"</span></span>:[{<span class="string"><span class="delimiter">"</span><span class="content">Effect</span><span class="delimiter">"</span></span>:<span class="string"><span class="delimiter">"</span><span class="content">Allow</span><span class="delimiter">"</span></span>,<span class="string"><span class="delimiter">"</span><span class="content">Principal</span><span class="delimiter">"</span></span>:{<span class="string"><span class="delimiter">"</span><span class="content">AWS</span><span class="delimiter">"</span></span>:[<span class="string"><span class="delimiter">"</span><span class="content">arn:aws:iam:::user/TESTER</span><span class="delimiter">"</span></span>]},<span class="string"><span class="delimiter">"</span><span class="content">Action</span><span class="delimiter">"</span></span>:[<span class="string"><span class="delimiter">"</span><span class="content">sts:AssumeRole</span><span class="delimiter">"</span></span>]}]}<span class="char">'}</span>}], {<span class="char">'R</span>equestId<span class="char">':</span> <span class="char">'t</span>x0000034b8fd6de729452c-<span class="octal">0063</span>ce4d91-<span class="integer">1175</span>-<span class="keyword">default</span><span class="char">'}</span>]
</span></code></pre></p>
<p>If you look closely at the RAW response you'll find the second "member" object doesn't have a preceding "{".</p> rgw - Backport #58327 (New): quincy: ListOpenIDConnectProviders XML format errorhttps://tracker.ceph.com/issues/583272022-12-20T19:42:51ZBackport Botrgw - Bug #57968 (New): Partial fix for XML responses returning different order of XML elementshttps://tracker.ceph.com/issues/579682022-11-03T14:13:58ZDaniel Iwan
<p>Hi<br />This is a follow up on original problem reported here<br /><a class="external" href="https://tracker.ceph.com/issues/52027">https://tracker.ceph.com/issues/52027</a></p>
<p>I've added my comment to that ticket but that may not get much attention since a fix to the issue has been merged.<br />Anyway, the fix is only partial.</p>
<p>I've added further explanation to the github conversation, with detailed description why the fix is only partial<br /><a class="external" href="https://github.com/ceph/ceph/pull/42683#issuecomment-1279475804">https://github.com/ceph/ceph/pull/42683#issuecomment-1279475804</a></p>
<p>I think that conversation is not tracked for the same reasons.</p>
<p>Any chance this problem is looked at again?</p>
<p>Regards<br />Daniel</p>