Ceph » rgw

from http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-24_03:05:03-rgw-master-distro-basic-gibba/6188142/remote/gibba029/log/rgw.ceph.client.0.log.gz:

2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault authentication method: token
2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault Secrets Engine: transit
2021-07-03T22:57:22.507+0000 7f7fb8032700  0 Loading Vault Token from filesystem
2021-07-03T22:57:22.507+0000 7f7fb8032700 20 Vault token file: /home/ubuntu/cephtest/vault-token
2021-07-03T22:57:22.507+0000 7f7fb8032700  0 ERROR: Vault token file '/home/ubuntu/cephtest/vault-token' not found
2021-07-03T22:57:22.507+0000 7f7fb8032700  5 req 4460 0.022999952s ERROR: failed to retrieve actual key from key_id: my-key-1

At first blush, this looks like a permissions problem.

The teuthology log claims that it wrote '/home/ubuntu/cephtest/vault-token'. If I remember right, the logic inside of ceph is not smart enough to report the actual error (EPERM vs. ENOENT) if the file open fails.

The easiest way to sort that out is to run a teuthology job with "interacive-on-error", wait until it hangs, then ssh into the node and inspect the situation. First step would be to see if the "ceph" user can actually read /home/ubuntu/cephtest/vault-token . Might need to store the vault token elsewhere or change permissions. Vault should be running, and the vault server binary is also the client: you can try "vault login" (with the root token), "vault status" (vault should be unsealed), and "vault lookup" (should display some info on the root token). My guess is this will prove to not require even that much knowledge of vault.