Ceph » rgw

When attempting to utilize the AssumeRoleWithWebIdentity STS API rgw_rest_sts.cc#L312 will attempt a HTTP GET to an improperly concatenated URL if the ISS from the JWT token ends in a slash (/). The result is a URL like https://myprovider//.well-known/openid-configuration which is not a well-formed URL and results in undefined behavior with different identity providers (case 1). There is also an alternative behavior when the configured OpenIDConnect provider does not contain the trailing / and the ISS does (case 2).

Keycloak - This is fine and the ISS does not contain a ending slash.
Authentik - The server will respond with a 301 redirect to the valid URL (without //) and RGWHTTPTransceiver does not follow redirects so request fails.

Source: https://github.com/ceph/ceph/blob/cca84e653dd5ea686884cb85fdd8e20703678274/src/rgw/rgw_rest_sts.cc#L312

Suggested Fix

Trim the trailing / from the ISS before concatenating the /.well-known/openid-configuration path suffix.

Logs

These are logs from the AssumeRoleWithWebIdentity API call. My logs are slightly altered for readability. This is also an offline test lab so I do not care that I am leaking credentials/identity.

(case 1) OpenIDConnect provider is configured with a URL of https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ and matches the ISS.

 1 ====== starting new request req=0x7efc3c0c8680 =====
   initializing for trans_id = tx000002dda8f6818905e64-0063fec66d-3607b8-default
   rgw api priority: s3=5 s3website=4
   host=s3.lab
   subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0
   final domain/bucket subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.lab s->info.request_uri=/
   get_handler handler=26RGWHandler_REST_Service_S3
   handler=26RGWHandler_REST_Service_S3
   getting op 4
   Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjI5MDc3ODY2YmNiMGQ3NWI3ZDJlNTFmZTQ1NDA1Yzk3IiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2xvZ2luLmxhYi9hcHBsaWNhdGlvbi9vL2Q3ZDY0NDk2ZTI2YzE1NmNhOWVhMDgwMmM1ZDdlZDFjLyIsInN1YiI6Im1hdGhldy51dHRlciIsImF1ZCI6InJhZG9zZ3ciLCJleHAiOjE2Nzc2NDMxMjMsImlhdCI6MTY3NzY0MTMyMywiYXV0aF90aW1lIjoxNjc3NjQxMzIzLCJhY3IiOiJnb2F1dGhlbnRpay5pby9wcm92aWRlcnMvb2F1dGgyL2RlZmF1bHQiLCJlbWFpbCI6Im1hdGhldy51dHRlckBzaGlmdDUuaW8iLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6Im1hdGhldy51dHRlciIsImdpdmVuX25hbWUiOiJtYXRoZXcudXR0ZXIiLCJmYW1pbHlfbmFtZSI6IiIsInByZWZlcnJlZF91c2VybmFtZSI6Im1hdGhldy51dHRlciIsIm5pY2tuYW1lIjoibWF0aGV3LnV0dGVyIiwiZ3JvdXBzIjpbImxhYnMtdXNlciIsInB2ZS11c2VyIiwiYXV0aGVudGlrIEFkbWlucyIsInMzLXVzZXIiXSwiY2lkIjoicmFkb3NndyIsInVpZCI6IlBvXn1FfGMzKmhxTHsxfU8_YUtzIS1nRTpDVzslMHYtcUlsZy01W2RwOXspb3VIcyxSOGE3bmt-ell2RkU8WzZCIUduaC4yMFQyQkBNJy10UylzQ2Q2dnJmJTtoLXwxMW95ZV4vLnVFTXo1NzJsRTcuSDBpdDBeJz58LWNNenopIn0.ZphoRcXj7gDfJl1zNhk5GBZvjcQlbWpcUp5GLOq5jJ50KVQG1c0zaKES7uqjC-AlNRS_r-Mp1x2a2kCqT_F_MgFKalfuajUif23g-kS8eVDEm16pdzW2_O_tKaOcI_BlBqu0izb-pXGc9TLrCclZ2n87ZGQZxKaJx2maGk7lpIV-IQ__b2ftxO3AewL1RSxOneAakXtCgZSd5ye-UgG5WKzBaIySeCvp1TaezZW_zvmMy-Z4EHMwkhkdzhOUJpvkFL_0FHsTQUV4Ws37CHinCvx6IbC9Mj2gBGf-aXbxT1aMPEuC9JsNmPavHKg9nqvwJ_RC7FycdJANQY0tMGrArppReSZprEywxuJmz1vUqNCdhPWrqLSoAxoOiaYPTVZPcuXs62TH5jeI9ysp2YVo_5iu0Tfsl7f27PxzdlH_PfgWzJt5Yit1T9xYPTbG98FlV3D8x4NJ1AF-_KGtqE8FbvKWsqs05qd24jyYEoutAiEReDySFrXal8uctGsVcmyxEg0cr7R9GsAMi4J3aDrYK7eNWWcjjlICM5qWfUHvkba-wtkHrIR-Rd_UmdyV1t9iixQzagfKjCQxRxpJKPtTz_OunaLeNZffE04xF2PxrFMGX0PoCSICE-hW4ODtDKa5Yq_omTLWNTGHc65mMQkIlmuD_R556mBbRbR7McamF98&RoleSessionName=mathew.utter&ProviderId=login.lab&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FAssumeRoleWithWebIdentityForOIDC
   get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.prerequest. state=0x7ef734005260 s->prefetch_data=0
   cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)
   sts:assume_role_web_identity scheduling with throttler client=0 cost=1
   sts:assume_role_web_identity op=31RGWSTSAssumeRoleWithWebIdentity
   sts:assume_role_web_identity verifying requester
   sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine
   sts:assume_role_web_identity  payload = {"iss":"https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/","sub":"mathew.utter","aud":"radosgw","exp":1677643123,"iat":1677641323,"auth_time":1677641323,"acr":"goauthentik.io/providers/oauth2/default","email":"mathew.utter@shift5.io","email_verified":true,"name":"mathew.utter","given_name":"mathew.utter","family_name":"","preferred_username":"mathew.utter","nickname":"mathew.utter","groups":["labs-user","pve-user","authentik Admins","s3-user"],"cid":"radosgw","uid":"Po^}E|c3*hqL{1}O?aKs!-gE:CW;%0v-qIlg-5[dp9{)ouHs,R8a7nk~zYvFE<[6B!Gnh.20T2B@M'-tS)sCd6vrf%;h-|11oye^/.uEMz572lE7.H0it0^'>|-cMzz)"}
   sts:assume_role_web_identity get_system_obj_state: rctx=0x7efc3c0c6b70 obj=default.rgw.meta:oidc:oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ state=0x7ef734005260 s->prefetch_data=0
   sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : hit (requested=0x6, cached=0x7)
   sts:assume_role_web_identity get_system_obj_state: s->obj_tag was set empty
   sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : hit (requested=0x1, cached=0x7)
20 sending request to https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c//.well-known/openid-configuration
20 register_request mgr=0x559fd8a118f0 req_data->id=4, curl_handle=0x7ef73400bcd0
20 link_request req_data=0x7ef734010550 req_data->id=4, curl_handle=0x7ef73400bcd0
   sts:assume_role_web_identity HTTP request res: -5
   sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13
   sts:assume_role_web_identity Failed the auth strategy, reason=-13
10 failed to authorize request
   op->ERRORHANDLER: err_no=-13 new_err_no=-13
   get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.postrequest. state=0x7ef790000f70 s->prefetch_data=0
   cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
   sts:assume_role_web_identity op status=0
   sts:assume_role_web_identity http status=403
 1 ====== req done req=0x7efc3c0c8680 op status=0 http_status=403 latency=0.028000288s ======

(case 2) OpenIDConnect provider is configured with a URL of https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c and does not match the ISS because of missing /.

 1 ====== starting new request req=0x7efc3c0c8680 =====
   initializing for trans_id = tx00000a51169179fd21dcc-0063fecf2a-3607b8-default
   rgw api priority: s3=5 s3website=4
   host=s3.lab
   subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0
   final domain/bucket subdomain= domain=s3.lab in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.lab s->info.request_uri=/
   get_handler handler=26RGWHandler_REST_Service_S3
   handler=26RGWHandler_REST_Service_S3
   getting op 4
   Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjI5MDc3ODY2YmNiMGQ3NWI3ZDJlNTFmZTQ1NDA1Yzk3IiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2xvZ2luLmxhYi9hcHBsaWNhdGlvbi9vL2Q3ZDY0NDk2ZTI2YzE1NmNhOWVhMDgwMmM1ZDdlZDFjLyIsInN1YiI6Im1hdGhldy51dHRlciIsImF1ZCI6InJhZG9zZ3ciLCJleHAiOjE2Nzc2NDUzNjAsImlhdCI6MTY3NzY0MzU2MCwiYXV0aF90aW1lIjoxNjc3NjQzNTYwLCJhY3IiOiJnb2F1dGhlbnRpay5pby9wcm92aWRlcnMvb2F1dGgyL2RlZmF1bHQiLCJlbWFpbCI6Im1hdGhldy51dHRlckBzaGlmdDUuaW8iLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6Im1hdGhldy51dHRlciIsImdpdmVuX25hbWUiOiJtYXRoZXcudXR0ZXIiLCJmYW1pbHlfbmFtZSI6IiIsInByZWZlcnJlZF91c2VybmFtZSI6Im1hdGhldy51dHRlciIsIm5pY2tuYW1lIjoibWF0aGV3LnV0dGVyIiwiZ3JvdXBzIjpbImxhYnMtdXNlciIsInB2ZS11c2VyIiwiYXV0aGVudGlrIEFkbWlucyIsInMzLXVzZXIiXSwiY2lkIjoicmFkb3NndyIsInVpZCI6Incja1t-Mm9bbDRKJF9TK3prPWxWLXw6b2A8P05yaVdvbWlcIiRiL2ItZi5ocCZGQT86UmM0R3p6K04zQSxEXCJ6VU5gfTxDbTkobjJhLkBeKTV7bk9-MTx3Wzpze1ZEPjo3UUduLjtyP3R5Jys7WFxcIX5TYFY_e2s5Myg7b3g3e3MqIn0.iPO-aTBtkTXwI8QDGWL0IbRwSAOwAVcofctVZpGATtQer59K8gCE0PlzBj_mMyd1Vge__W5HCrorfQNkDnI5ekvToc3tjsptY0gTAAFEfTFrmK7thLoxAsawRzcxxsmLFjHO8E0i2it-OLITMVucVzi5kKobUs5uR7TFZLHe39yslI2Ux3z6iBMe7Pb6eSZh36xiQZ7-mHFSZu05Zt6j8rg8yB9k0ckZZg8uQwhp8-E5KHdmkzUaWpldCHI73XiYH7gZVT3mJgFvAhLMFvhr96kgOT0cUKuNx3iQBChV7c-1_mlcDYbQkuZfvzlSqGAa0tdBMSX13Q9cOgw4i0S9i7ApGwyY5C5PXaqOTIgkEB91hcziUZuiWisT5BLFbgb-Mv4OmU1iA4w26a9Jl4bdtY_KPJwMkHZfOW_WqB27vZSG_DwInrqSXaMBr-mUU3sPralrIBF75WhmcY8iNRik136oXUka3WiJLcG4hTIT8AwSziISTdzyqS9nDL9OJkCoivZrjZuFLffbhHV2NXJt3bUQjEZqzpkznmyVvNYabubqm-rG0-Nu8czuf5MlxdmiqDCB0JnpVJE0XeGie0hWg1TJbrJe0N3z3EYs__82mSyMg5ifG_H0QTWzOvx2SKfNLE74kpgUYlwTWBth-kZPp0rUm7spiBurKXD3AKpXWx4&RoleSessionName=mathew.utter&ProviderId=login.lab&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FAssumeRoleWithWebIdentityForOIDC
   get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.prerequest. state=0x7ef618004500 s->prefetch_data=0
   cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)
   sts:assume_role_web_identity scheduling with throttler client=0 cost=1
   sts:assume_role_web_identity op=31RGWSTSAssumeRoleWithWebIdentity
   sts:assume_role_web_identity verifying requester
   sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine
   sts:assume_role_web_identity  payload = {"iss":"https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/","sub":"mathew.utter","aud":"radosgw","exp":1677645360,"iat":1677643560,"auth_time":1677643560,"acr":"goauthentik.io/providers/oauth2/default","email":"mathew.utter@shift5.io","email_verified":true,"name":"mathew.utter","given_name":"mathew.utter","family_name":"","preferred_username":"mathew.utter","nickname":"mathew.utter","groups":["labs-user","pve-user","authentik Admins","s3-user"],"cid":"radosgw","uid":"w#k[~2o[l4J$_S+zk=lV-|:o`<?NriWomi\"$b/b-f.hp&FA?:Rc4Gzz+N3A,D\"zUN`}<Cm9(n2a.@^)5{nO~1<w[:s{VD>:7QGn.;r?ty'+;X\\!~S`V?{k93(;ox7{s*"}
   sts:assume_role_web_identity get_system_obj_state: rctx=0x7efc3c0c6b70 obj=default.rgw.meta:oidc:oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ state=0x7ef618004500 s->prefetch_data=0
   sts:assume_role_web_identity cache get: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ : miss
   sts:assume_role_web_identity WARNING: blocking librados call
 1 -- 172.25.1.102:0/636600742 --> [v2:172.25.1.102:6800/1514588,v1:172.25.1.102:6801/1514588] -- osd_op(unknown.0.0:9038 6.a 6:512c5d0d:oidc::oidc_url.login.lab%2fapplication%2fo%2fd7d64496e26c156ca9ea0802c5d7ed1c%2f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e1115) v8 -- 0x7ef61800d9c0 con 0x7efc2c02f8a0
 1 -- 172.25.1.102:0/636600742 <== osd.13 v2:172.25.1.102:6800/1514588 1915 ==== osd_op_reply(9038 oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 252+0+0 (crc 0 0 0) 0x7efc2c105040 con 0x7efc2c02f8a0
   sts:assume_role_web_identity cache put: name=default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ info.flags=0x0
   sts:assume_role_web_identity adding default.rgw.meta+oidc+oidc_url.login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/ to cache LRU end
   sts:assume_role_web_identity Couldn't get oidc provider info using input isshttps://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/
   sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13
   sts:assume_role_web_identity Failed the auth strategy, reason=-13
10 failed to authorize request
   op->ERRORHANDLER: err_no=-13 new_err_no=-13
   get_system_obj_state: rctx=0x7efc3c0c76b0 obj=default.rgw.log:script.postrequest. state=0x7ef618004500 s->prefetch_data=0
   cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
   sts:assume_role_web_identity op status=0
   sts:assume_role_web_identity http status=403
 1 ====== req done req=0x7efc3c0c8680 op status=0 http_status=403 latency=0.000000000s ======

No PR at the moment.