Bug #54247: rgw/crypt/barbican: Cannot create secret - rgw - Ceph

Actions

Copy link

Bug #54247

closed

rgw/crypt/barbican: Cannot create secret

Added by Casey Bodley about 2 years ago. Updated 8 months ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Target version:

% Done:

100%

Source:

Tags:

barbican backport_processed

Backport:

pacific quincy

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

45379

Crash signature (v1):

Crash signature (v2):

Description

this started failing recently. http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-10_03:05:03-rgw-master-distro-default-smithi/6672411/teuthology.log

2022-02-10T04:12:03.567 INFO:tasks.barbican.client.0:Started
2022-02-10T04:12:03.567 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest/barbican && . .barbicanenv/bin/activate && sleep 15
2022-02-10T04:12:04.806 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:04.808 51514 INFO barbican.model.repositories [-] Setting up database engine and session factory[00m
2022-02-10T04:12:04.815 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:04.818 51514 INFO barbican.model.repositories [-] Auto-creating barbican registry DB[00m
2022-02-10T04:12:04.936 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:04.937 51514 WARNING barbican.model.migration.commands [-] !!! Limited support for migration commands using sqlite databases; This operation may not succeed.[00m
2022-02-10T04:12:04.940 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:04.942 51514 INFO alembic.runtime.migration [-] Context impl SQLiteImpl.[00m
2022-02-10T04:12:04.940 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:04.942 51514 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.[00m
2022-02-10T04:12:05.027 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.029 51514 INFO alembic.runtime.migration [-] Running stamp_revision  -> 39cf2e645cba[00m
2022-02-10T04:12:05.079 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.081 51514 WARNING barbican.api.controllers.cas [-] Deprecated: Certificate Authorities API has been deprecated in the Newton release. It will be removed in the Pike release.[00m
2022-02-10T04:12:05.220 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.222 51514 INFO barbican.api.app [-] Barbican app created and initialized[00m
2022-02-10T04:12:05.222 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.224 51514 WARNING keystonemiddleware._common.config [-] The option "auth_url" is not known to keystonemiddleware[00m
2022-02-10T04:12:05.222 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.224 51514 WARNING keystonemiddleware._common.config [-] The option "username" is not known to keystonemiddleware[00m
2022-02-10T04:12:05.222 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.224 51514 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" is not known to keystonemiddleware[00m
2022-02-10T04:12:05.223 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.224 51514 WARNING keystonemiddleware._common.config [-] The option "password" is not known to keystonemiddleware[00m
2022-02-10T04:12:05.223 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:05.224 51514 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.[00m
2022-02-10T04:12:18.582 INFO:tasks.barbican:barbican_url=http://smithi196.front.sepia.ceph.com:9311
2022-02-10T04:12:19.148 INFO:tasks.barbican.client.0.smithi196.stdout:serving on 0.0.0.0:9311 view at http://127.0.0.1:9311
2022-02-10T04:12:19.149 INFO:tasks.barbican.client.0.smithi196.stdout:2022-02-10 04:12:19.151 51514 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use  memcache for the auth_token token cache by setting the memcached_servers option.[00m
2022-02-10T04:12:19.579 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_7cebb3f2319fd6c8340c0f7cd15a137e747fd32e/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph_f5b79d7e6bbd3fd92c91375c16357753c45cf8aa/qa/tasks/barbican.py", line 371, in create_secrets
    raise Exception("Cannot create secret")

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Casey Bodley about 2 years ago

Assignee set to Marcus Watts

Actions

Copy link

Updated by Kefu Chai about 2 years ago

/a/kchai-2022-03-13_10:17:16-rgw-wip-rapidjson-kefu-distro-default-smithi/6734938/

Actions

Copy link

Updated by Casey Bodley over 1 year ago

scripts to deploy keystone/barbican:
https://github.com/ceph/ceph/blob/main/qa/tasks/barbican.py
https://github.com/ceph/ceph/blob/main/qa/tasks/keystone.py

configuration for keystone+barbican:
https://github.com/ceph/ceph/blob/main/qa/suites/rgw/crypt/2-kms/barbican.yaml

Actions

Copy link

Updated by Tobias Urdin over 1 year ago

I will see if we can get some attention on this, see my reply in ceph-users mailing list. I've proposed [1] way back to bump versions but that's probably not enought, there is probably room for more improvements as well.

[1] https://github.com/ceph/ceph/pull/45379

Actions

Copy link

Updated by Tobias Urdin over 1 year ago

Based on the linked log post it seems Barbican is actually started based on the logs but never receives the request (cannot see the request in the output). So it fails the check [1].

Could this simply be a race condition between Barbican starting and Teuthology running the create_secrets code? I can't see the response code being logged that would probably help.

[1] https://github.com/ceph/ceph/blob/main/qa/tasks/barbican.py#L369

Actions

Copy link

Updated by Casey Bodley over 1 year ago

Status changed from New to In Progress
Pull request ID set to 47605

thanks Tobias, these really do look like startup races. i'm testing a fix in https://github.com/ceph/ceph/pull/47605 that replaces 'sleep 15' with a loop that polls the http endpoint with curl

Actions

Copy link

Updated by Casey Bodley over 1 year ago

Casey Bodley wrote:

thanks Tobias, these really do look like startup races. i'm testing a fix in https://github.com/ceph/ceph/pull/47605 that replaces 'sleep 15' with a loop that polls the http endpoint with curl

i guess there's more to it than just the startup race - in testing, i see that the key creation response is a 500: