Project

General

Profile

Bug #51539

rgw/crypt s3tests with vault: Failed to retrieve the actual key, kms-keyid: my-key-1

Added by Casey Bodley over 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
vault sse
Backport:
octopus pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

several s3test cases under the rgw/crypt suite are failing with this error "Failed to retrieve the actual key, kms-keyid: my-key-1"

ex. http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-24_03:05:03-rgw-master-distro-basic-gibba/6188142/teuthology.log from https://pulpito.ceph.com/teuthology-2021-06-24_03:05:03-rgw-master-distro-basic-gibba/

2021-07-03T22:59:00.278 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_method_head
2021-07-03T22:59:00.303 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_present
2021-07-03T22:59:00.327 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload
2021-07-03T22:59:00.352 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1
2021-07-03T22:59:00.377 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2
2021-07-03T22:59:00.401 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b
2021-07-03T22:59:00.427 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb
2021-07-03T22:59:00.452 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB
2021-07-03T22:59:00.476 INFO:teuthology.orchestra.run.gibba029.stderr:ERROR: s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b

Related issues

Copied to rgw - Backport #53098: pacific: rgw/crypt s3tests with vault: Failed to retrieve the actual key, kms-keyid: my-key-1 Resolved
Copied to rgw - Backport #53099: octopus: rgw/crypt s3tests with vault: Failed to retrieve the actual key, kms-keyid: my-key-1 Resolved

History

#1 Updated by Casey Bodley over 1 year ago

from http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-24_03:05:03-rgw-master-distro-basic-gibba/6188142/remote/gibba029/log/rgw.ceph.client.0.log.gz:

2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault authentication method: token
2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault Secrets Engine: transit
2021-07-03T22:57:22.507+0000 7f7fb8032700  0 Loading Vault Token from filesystem
2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault token file: /home/ubuntu/cephtest/vault-token
2021-07-03T22:57:22.507+0000 7f7fb8032700  0 ERROR: Vault token file '/home/ubuntu/cephtest/vault-token' not found
2021-07-03T22:57:22.507+0000 7f7fb8032700  5 req 4460 0.022999952s ERROR: failed to retrieve actual key from key_id: my-key-1

#2 Updated by Casey Bodley over 1 year ago

  • Assignee set to Marcus Watts

#3 Updated by Marcus Watts over 1 year ago

At first blush, this looks like a permissions problem.

The teuthology log claims that it wrote '/home/ubuntu/cephtest/vault-token'. If I remember right, the logic inside of ceph is not smart enough to report the actual error (EPERM vs. ENOENT) if the file open fails.

The easiest way to sort that out is to run a teuthology job with "interacive-on-error", wait until it hangs, then ssh into the node and inspect the situation. First step would be to see if the "ceph" user can actually read /home/ubuntu/cephtest/vault-token . Might need to store the vault token elsewhere or change permissions. Vault should be running, and the vault server binary is also the client: you can try "vault login" (with the root token), "vault status" (vault should be unsealed), and "vault lookup" (should display some info on the root token). My guess is this will prove to not require even that much knowledge of vault.

#4 Updated by Casey Bodley about 1 year ago

  • Status changed from New to Pending Backport
  • Backport set to octopus pacific

#5 Updated by Backport Bot about 1 year ago

  • Copied to Backport #53098: pacific: rgw/crypt s3tests with vault: Failed to retrieve the actual key, kms-keyid: my-key-1 added

#6 Updated by Backport Bot about 1 year ago

  • Copied to Backport #53099: octopus: rgw/crypt s3tests with vault: Failed to retrieve the actual key, kms-keyid: my-key-1 added

#7 Updated by Cory Snyder about 1 year ago

  • Pull request ID set to 43625

#8 Updated by Loïc Dachary 11 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF