Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2024-02-12T07:32:26ZCeph
Redmine rgw - Cleanup #64388 (New): improve error message when expired pre-signed urls are being usedhttps://tracker.ceph.com/issues/643882024-02-12T07:32:26ZPritha Srivastavaprsrivas@redhat.com
<p>improve the error message that is logged and that is returned back to the client, in case an expired pre-signed URL is used.</p> rgw - Feature #63214 (New): rgw: update thumbprint list of an oidc providerhttps://tracker.ceph.com/issues/632142023-10-16T07:51:02ZPritha Srivastavaprsrivas@redhat.com
<p>the implementation of oidc provider in rgw currently doesnt support updating thumbprint list of an existing oidc provider.<br />this tracker tracks implementation of the same.</p> rgw - Feature #63213 (New): rgw:add client id to open id connect providerhttps://tracker.ceph.com/issues/632132023-10-16T07:49:31ZPritha Srivastavaprsrivas@redhat.com
<p>The implementation of openid connect provider in rgw currently does not provide the feature to add client id to an existing openid connect provider.<br />This tracker tracks the implementation of the same.</p> rgw - Bug #62541 (Triaged): docs: sts-AssumeRoleWithWebIdentity does not work for tenanted roleshttps://tracker.ceph.com/issues/625412023-08-23T07:09:42ZGuenter Sandner
<p>having two roles using the same assume-role-policy-document defined like this<br /><pre>
# create non-tenanted role
radosgw-admin role create --role-name='devS3Access' --path=/ \
--assume-role-policy-doc='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/192.168.100.69:9080/realms/aurora-dev"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"192.168.100.69:9080/realms/aurora-dev:azp":"aurora-api-gateway-dev"}}}]}'
# create tenanted role
radosgw-admin role create --tenant tenant1 --role-name='tenant1S3Access' --path=/ \
--assume-role-policy-doc='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/192.168.100.69:9080/realms/aurora-dev"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"192.168.100.69:9080/realms/aurora-dev:azp":"aurora-api-gateway-dev"}}}]}'
</pre><br />calling "sts asume-role-with-web-identity" works for the non tenanted role "devS3Access", but not for the tenanted role "tenant1S3Access"</p>
<pre>
[root@rook-ceph-tools-855599bf84-wpwhr /]# aws --endpoint=http://$AWS_HOST:$PORT sts assume-role-with-web-identity \
> --role-arn 'arn:aws:iam::tenant1:role/S3Access' \
> --role-session-name 'tenant1S3Access' \
> --web-identity-token "${ID_TOKEN_DEV}" \
> --duration-seconds 3600 >tenant1_secrets || echo FAILED
An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown
</pre>
<p>in the logs we can see this for the non-tenanted role:<br /><pre>
debug 2023-08-22T11:28:14.541+0000 7f2b2359c700 10 req 18142818711300298359 0.000000000s sts:assume_role_web_identity cache get: name=my-store.rgw.meta+oidc+oidc_url.192.168.100.69:9080/realms/aurora-dev : hit (requested=0x1, cached=0x7)
</pre><br />but this for the tenanted role:<br /><pre>
ebug 2023-08-22T11:23:44.114+0000 7f2b78646700 10 req 13090842789017334974 0.001000009s sts:assume_role_web_identity cache get: name=my-store.rgw.meta+oidc+tenant1oidc_url.192.168.100.69:9080/realms/aurora-dev : hit (negative entry)
debug 2023-08-22T11:23:44.114+0000 7f2b78646700 0 req 13090842789017334974 0.001000009s sts:assume_role_web_identity Couldn't get oidc provider info using input isshttp://192.168.100.69:9080/realms/aurora-dev
</pre></p> rgw - Bug #61916 (Pending Backport): qa/sts: test_list_buckets_invalid_auth and test_list_buckets...https://tracker.ceph.com/issues/619162023-07-06T13:54:16ZCasey Bodleycbodley@redhat.com
<p>until <a class="external" href="https://github.com/ceph/ceph/pull/52216">https://github.com/ceph/ceph/pull/52216</a>, we <strong>only</strong> ran the test cases marked <code>sts_tests</code> and <code>webidentity_tests</code> in this configuration. now that we run all test cases, these two ListBuckets tests are failing with the wrong error code</p>
<p>ex. <a class="external" href="http://qa-proxy.ceph.com/teuthology/cbodley-2023-06-30_13:14:21-rgw-main-distro-default-smithi/7322679/teuthology.log">http://qa-proxy.ceph.com/teuthology/cbodley-2023-06-30_13:14:21-rgw-main-distro-default-smithi/7322679/teuthology.log</a></p> rgw - Backport #61633 (New): quincy: failure of cls_rgw_gc_list is not handledhttps://tracker.ceph.com/issues/616332023-06-09T17:26:46ZBackport Botrgw - Backport #61631 (New): reef: failure of cls_rgw_gc_list is not handledhttps://tracker.ceph.com/issues/616312023-06-09T17:26:31ZBackport Botrgw - Backport #59607 (New): reef: Renaming large files fails with 403 when using temporary creds...https://tracker.ceph.com/issues/596072023-05-02T20:59:46ZBackport Botrgw - Bug #58628 (Pending Backport): Renaming large files fails with 403 when using temporary cre...https://tracker.ceph.com/issues/586282023-02-02T04:20:33ZPritha Srivastavaprsrivas@redhat.com
<p>Renaming large files fails with 403 when using temporary creds returned by STS. These files are uploaded using MultipartUpload. aws s3 cli uses 8MB as the limit and s3cmd uses 5MB.</p> rgw - Backport #57364 (In Progress): quincy: multisite: metadata sync does not sync STS metadata ...https://tracker.ceph.com/issues/573642022-09-01T05:04:22ZPritha Srivastavaprsrivas@redhat.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/48030">https://github.com/ceph/ceph/pull/48030</a></p> rgw - Bug #57323 (Pending Backport): failure of cls_rgw_gc_list is not handledhttps://tracker.ceph.com/issues/573232022-08-29T08:42:29ZYuval Lifshitzyuvalif@yahoo.com
<p>in "RGWGC::process()" the return code from cls_rgw_gc_list is ignored.<br />one possible issue is that the value of "truncated' is used uninitialized in ldout<br />coverity CID 1512008</p> rgw - Bug #54562 (New): Validation of a token signature may use a wrong certificatehttps://tracker.ceph.com/issues/545622022-03-15T11:49:36ZDaniel Iwan
<p>This is a follow up bug report for issue originally created in Ceph project rather than in rgw<br />See <a class="external" href="https://tracker.ceph.com/issues/54239">https://tracker.ceph.com/issues/54239</a></p>
<p>It does not look original issue has been picked up/triaged and I cannot change the project<br />Apologies for double posting<br />Feel free to delete the original one</p>
<p>Details of the bug</p>
<p>When RGW STS validates the token and OIDC provider returns more than 1 key, incorrect certificate will be picked to validate the token.<br />Reproduced with Keycloak v16.1.1</p>
<p>The bug is in WebTokenEngine::validate_signature()<br /><a class="external" href="https://github.com/ceph/ceph/blob/f5b79d7e6bbd3fd92c91375c16357753c45cf8aa/src/rgw/rgw_rest_sts.cc#L377">https://github.com/ceph/ceph/blob/f5b79d7e6bbd3fd92c91375c16357753c45cf8aa/src/rgw/rgw_rest_sts.cc#L377</a></p>
<p>The code looks as follows</p>
<pre>
vector<string> x5c;
if (JSONDecoder::decode_json("x5c", x5c, &parser)) {
string cert;
bool found_valid_cert = false;
for (auto& it : x5c) {
cert = "-----BEGIN CERTIFICATE-----\n" + it + "\n-----END CERTIFICATE-----";
ldpp_dout(dpp, 20) << "Certificate is: " << cert.c_str() << dendl;
if (is_cert_valid(thumbprints, cert)) {
found_valid_cert = true;
break;
}
found_valid_cert = true;
}
if (! found_valid_cert) {
ldpp_dout(dpp, 0) << "Cert doesn't match that with the thumbprints registered with oidc provider: " << cert.c_str() << dendl;
throw -EINVAL;
}
</pre>
<p>it seems second</p>
<p>found_valid_cert = true;</p>
<p>is not needed there</p>
<p>Also I'm not sure if</p>
<p>JSONDecoder::decode_json("x5c", x5c, &parser)</p>
<p>picks only first element matching x5c or all of them.<br />Regardless, it does not seem like it will set the right certificate in the cert variable.<br />The first one fetched from the URL may actually be for encryption rather than for signature (see my log example)</p>
<p>In my case logs shows (some values have been trimmed here) only first certificate with "use":"enc" is printed.<br />Right after that failure to validate the signature.</p>
<pre>
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 10 req 14396669455332211639 0.011999830s sts:assume_role_web_identity cache put: name=something.rgw.meta+oidc+555453oidc_url.localhost.ceph-vm.com:8443/auth/realms/555453 info.flags=0x1
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 10 req 14396669455332211639 0.011999830s sts:assume_role_web_identity moving something.rgw.meta+oidc+555453oidc_url.localhost.ceph-vm.com:8443/auth/realms/555453 to cache LRU end
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 sending request to https://localhost.ceph-vm.com:8443/auth/realms/555453/protocol/openid-connect/certs
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 ssl verification is set to off
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 register_request mgr=0x55c60518b9e0 req_data->id=0, curl_handle=0x55c606547800
debug 2022-02-09T18:39:06.221+0000 7f026750a700 20 link_request req_data=0x55c605f12960 req_data->id=0, curl_handle=0x55c606547800
debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity HTTP status: 200
debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity JSON Response is: {"keys":[ {"kid":"tVTy-HaUNNAgjIi_gsEKa_HwgmOs8lLQ39hGuO8RWNM","kty":"RSA","alg":"RSA-OAEP","use":"enc","n":"22UW2Te8z_iidibSKsO95pvk3qYysPjXMwdErX25grTVM9FcQq8b11aU3T_b9yOKGiV_oL2soXb1nJfobbgBIi8FZ7BujQgaMVlpYAhcmwl9BgVwTPEblw0YmhqA6Q2HsWvq1zY...","e":"AQAB","x5c":["MIICmzCCAYMCBgF+36gdgDANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDbZRbZN7zP..."],"x5t":"Sm_1xvmsTFvOv8QiiAHTpcPHc2A","x5t#S256":"zKVKLsHGg6zKV3RYfV4nb2iO4zJU4lE2H65MNgk4mgM"}, {"kid":"on4WZBMyEEEidTjR2p31HA_PeZM3Pol1AWcSWq2eqb4","kty":"RSA","alg":"RS256","use":"sig","n":"tqKD_fHkGS9Bb8qWaDNpiJHEnzEUJvlVps0XWvf1s0JjBECWHfCb7X_AyLlfHmcBQJ2NWK_ztfEhYCP_9jzaQqYTx...","e":"AQAB","x5c":["MIICmzCCAYMCBgF+36gcsTANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2ooP98eQZL0FvypZoM2mIkcSfMRQm..."],"x5t":"Pcu3M4E-Ynw2IicqIBqOOoawmck","x5t#S256":"3GWgoMuAniK5_LjlsT73RHKvL3zllENb7xK5MtiK2R0"}]}
debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity Certificate is: -----BEGIN CERTIFICATE-----
MIICmzCCAYMCBgF+36gdgDANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDbZRbZN7zP...
-----END CERTIFICATE-----
debug 2022-02-09T18:39:06.309+0000 7f0235ca7700 0 req 14396669455332211639 0.099998586s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
debug 2022-02-09T18:39:06.309+0000 7f0235ca7700 20 req 14396669455332211639 0.099998586s sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13
</pre>
<p>This has been reproduced in Ceph 16.2.7 but applies to master as well.</p>
<p>Regards<br />Daniel</p> rgw - Bug #53512 (Pending Backport): radosgw-admin bucket rm --bucket=${bucket} --bypass-gc --pur...https://tracker.ceph.com/issues/535122021-12-07T15:26:48ZPritha Srivastavaprsrivas@redhat.com
<p>radosgw-admin bucket rm --bucket=testbucket1 --bypass-gc --purge-objects crashes</p>
<p>Steps to Reproduce:<br />1. Create a bucket :<br />s3cmd mb s3://testbucket1</p>
<p>2. Abort the multipart upload :</p>
<p>s3cmd put /boot/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img s3://testbucket1<br />upload: '/boot/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' -> 's3://testbucket1/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' [part 1 of 6, 15MB] [1 of 1]<br /> 15728640 of 15728640 100% in 0s 21.19 MB/s done<br />upload: '/boot/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' -> 's3://testbucket1/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' [part 2 of 6, 15MB] [1 of 1]<br /> 15728640 of 15728640 100% in 0s 22.57 MB/s done<br />upload: '/boot/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' -> 's3://testbucket1/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' [part 3 of 6, 15MB] [1 of 1]<br /> 65536 of 15728640 0% in 0s 1301.21 kB/s^CERROR: <br />Upload of '/boot/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img' part 3 failed. Use<br /> /usr/bin/s3cmd abortmp s3://testbucket1/initramfs-0-rescue-6d298da93b6e47828ddf0d55f92b4461.img 2~eltPdbtxJhLeQmK2d0uvgBeXjS6kDvB<br />to abort the upload, or<br /> /usr/bin/s3cmd --upload-id 2~eltPdbtxJhLeQmK2d0uvgBeXjS6kDvB put ...<br />to continue the upload.<br />See ya!</p>
3. Run the following command to remove the bucket :
<ol>
<li>radosgw-admin bucket rm --bucket=testbucket1 --bypass-gc --purge-objects</li>
</ol> rgw - Bug #51068 (Pending Backport): multisite: metadata sync does not sync STS metadata (e.g., r...https://tracker.ceph.com/issues/510682021-06-02T14:44:16ZMatt Benjaminmbenjamin@redhat.com
<p>Support for synchronizing roles/role policy/oidc provider config (etc?) is required to easily use STS AA in replicated setups.</p> Ceph - Bug #23831 (Triaged): bucket policy ipdeny not in effecthttps://tracker.ceph.com/issues/238312018-04-24T02:41:19ZAmine Liu
<p>I set bucket policy ipdeny, but still can read and write objects; but only set versioning will return 403;</p>