Ceph » rgw

This is a follow up bug report for issue originally created in Ceph project rather than in rgw
See https://tracker.ceph.com/issues/54239

It does not look original issue has been picked up/triaged and I cannot change the project
Apologies for double posting
Feel free to delete the original one

Details of the bug

When RGW STS validates the token and OIDC provider returns more than 1 key, incorrect certificate will be picked to validate the token.
Reproduced with Keycloak v16.1.1

The bug is in WebTokenEngine::validate_signature()
https://github.com/ceph/ceph/blob/f5b79d7e6bbd3fd92c91375c16357753c45cf8aa/src/rgw/rgw_rest_sts.cc#L377

The code looks as follows

vector<string> x5c;
if (JSONDecoder::decode_json("x5c", x5c, &parser)) {
string cert;
bool found_valid_cert = false;
for (auto& it : x5c) {
cert = "-----BEGIN CERTIFICATE-----\n" + it + "\n-----END CERTIFICATE-----";
ldpp_dout(dpp, 20) << "Certificate is: " << cert.c_str() << dendl;
if (is_cert_valid(thumbprints, cert)) {
found_valid_cert = true;
break;
}
found_valid_cert = true;
}
if (! found_valid_cert) {
ldpp_dout(dpp, 0) << "Cert doesn't match that with the thumbprints registered with oidc provider: " << cert.c_str() << dendl;
throw -EINVAL;
}

it seems second

found_valid_cert = true;

is not needed there

Also I'm not sure if

JSONDecoder::decode_json("x5c", x5c, &parser)

picks only first element matching x5c or all of them.
Regardless, it does not seem like it will set the right certificate in the cert variable.
The first one fetched from the URL may actually be for encryption rather than for signature (see my log example)

In my case logs shows (some values have been trimmed here) only first certificate with "use":"enc" is printed.
Right after that failure to validate the signature.

debug 2022-02-09T18:39:06.221+0000 7f02364a8700 10 req 14396669455332211639 0.011999830s sts:assume_role_web_identity cache put: name=something.rgw.meta+oidc+555453oidc_url.localhost.ceph-vm.com:8443/auth/realms/555453 info.flags=0x1
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 10 req 14396669455332211639 0.011999830s sts:assume_role_web_identity moving something.rgw.meta+oidc+555453oidc_url.localhost.ceph-vm.com:8443/auth/realms/555453 to cache LRU end
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 sending request to https://localhost.ceph-vm.com:8443/auth/realms/555453/protocol/openid-connect/certs
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 ssl verification is set to off
debug 2022-02-09T18:39:06.221+0000 7f02364a8700 20 register_request mgr=0x55c60518b9e0 req_data->id=0, curl_handle=0x55c606547800
debug 2022-02-09T18:39:06.221+0000 7f026750a700 20 link_request req_data=0x55c605f12960 req_data->id=0, curl_handle=0x55c606547800
debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity HTTP status: 200
debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity JSON Response is: {"keys":[ {"kid":"tVTy-HaUNNAgjIi_gsEKa_HwgmOs8lLQ39hGuO8RWNM","kty":"RSA","alg":"RSA-OAEP","use":"enc","n":"22UW2Te8z_iidibSKsO95pvk3qYysPjXMwdErX25grTVM9FcQq8b11aU3T_b9yOKGiV_oL2soXb1nJfobbgBIi8FZ7BujQgaMVlpYAhcmwl9BgVwTPEblw0YmhqA6Q2HsWvq1zY...","e":"AQAB","x5c":["MIICmzCCAYMCBgF+36gdgDANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDbZRbZN7zP..."],"x5t":"Sm_1xvmsTFvOv8QiiAHTpcPHc2A","x5t#S256":"zKVKLsHGg6zKV3RYfV4nb2iO4zJU4lE2H65MNgk4mgM"}, {"kid":"on4WZBMyEEEidTjR2p31HA_PeZM3Pol1AWcSWq2eqb4","kty":"RSA","alg":"RS256","use":"sig","n":"tqKD_fHkGS9Bb8qWaDNpiJHEnzEUJvlVps0XWvf1s0JjBECWHfCb7X_AyLlfHmcBQJ2NWK_ztfEhYCP_9jzaQqYTx...","e":"AQAB","x5c":["MIICmzCCAYMCBgF+36gcsTANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2ooP98eQZL0FvypZoM2mIkcSfMRQm..."],"x5t":"Pcu3M4E-Ynw2IicqIBqOOoawmck","x5t#S256":"3GWgoMuAniK5_LjlsT73RHKvL3zllENb7xK5MtiK2R0"}]}

debug 2022-02-09T18:39:06.305+0000 7f0235ca7700 20 req 14396669455332211639 0.095998637s sts:assume_role_web_identity Certificate is: -----BEGIN CERTIFICATE-----
MIICmzCCAYMCBgF+36gdgDANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAY1NTU0NTMwHhcNMjIwMjA5MTgwMjIzWhcNMzIwMjA5MTgwNDAzWjARMQ8wDQYDVQQDDAY1NTU0NTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDbZRbZN7zP...
-----END CERTIFICATE-----
debug 2022-02-09T18:39:06.309+0000 7f0235ca7700 0 req 14396669455332211639 0.099998586s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
debug 2022-02-09T18:39:06.309+0000 7f0235ca7700 20 req 14396669455332211639 0.099998586s sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13

This has been reproduced in Ceph 16.2.7 but applies to master as well.

Regards
Daniel