Bug #53090: rgw/sts: 403 response seen from radosgw on cleanup from (passed!) s3-tests run - rgw - Ceph

Actions

Copy link

Bug #53090

closed

rgw/sts: 403 response seen from radosgw on cleanup from (passed!) s3-tests run

Added by Matt Benjamin over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Kalpesh Pandya

Target version:

% Done:

Source:

Tags:

sts test

Backport:

Regression:

Yes

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

428

Crash signature (v1):

Crash signature (v2):

Description

From teuthology.log:

2021-10-28T15:19:49.750 INFO:teuthology.orchestra.run.smithi105.stderr:botocore.parsers: DEBUG: Response body:
2021-10-28T15:19:49.750 INFO:teuthology.orchestra.run.smithi105.stderr:b'<Error><Code>AccessDenied</Code><RequestId>tx000007c970358ca5f4ac4-00617abf95-11a6-default</RequestId><HostId>11a6-default-default</HostId></Error>'
2021-10-28T15:19:49.750 INFO:teuthology.orchestra.run.smithi105.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x7f2006dd6dc0>
2021-10-28T15:19:49.751 INFO:teuthology.orchestra.run.smithi105.stderr:botocore.retryhandler: DEBUG: No retry needed.
2021-10-28T15:19:49.751 INFO:teuthology.orchestra.run.smithi105.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f2006dd6fd0>>
2021-10-28T15:19:49.751 INFO:teuthology.orchestra.run.smithi105.stderr:--------------------- >> end captured logging << ---------------------
2021-10-28T15:19:49.751 INFO:teuthology.orchestra.run.smithi105.stderr:
2021-10-28T15:19:49.752 INFO:teuthology.orchestra.run.smithi105.stderr:----------------------------------------------------------------------
2021-10-28T15:19:49.752 INFO:teuthology.orchestra.run.smithi105.stderr:Ran 7 tests in 901.267s
2021-10-28T15:19:49.752 INFO:teuthology.orchestra.run.smithi105.stderr:
2021-10-28T15:19:49.752 INFO:teuthology.orchestra.run.smithi105.stderr:FAILED (errors=1)
2021-10-28T15:19:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-10-28T15:19:49.782 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/contextutil.py", line 31, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_72f536ef36dbc5c016fc4968ae47950069a9673f/qa/tasks/s3tests.py", line 443, in run_tests
label="s3 tests against rgw"
File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi105 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts'
2021-10-28T15:19:49.783 INFO:tasks.s3tests:Cleaning up boto...
2021-10-28T15:19:49.784 DEBUG:teuthology.orchestra.run.smithi105:> rm /home/ubuntu/cephtest/boto.cfg

From #rh-rgw:
(12:01:49 PM) cbodley: hey mbenjami, i'm looking at the log too
(12:02:56 PM) cbodley: the test cases passed, but the cleanup code got an error trying to list buckets..
(12:03:05 PM) cbodley: "GET / HTTP/1.1" fails with 403
(12:03:24 PM) mbenjamin: that doesn't seem like something the ops-log change could have caused
(12:05:30 PM) cbodley: looking at the rgw log, s3:list_buckets rgw::auth::s3::AWSAuthStrategy granted access
(12:05:42 PM) cbodley: s3:list_buckets verifying op permissions
(12:05:43 PM) cbodley: op->ERRORHANDLER: err_no=-13 new_err_no=-13
(12:05:53 PM) jbautista|gone is now known as jbautista
(12:06:11 PM) cbodley: so somehow authentication succeeded, but authorization to list buckets failed?
(12:07:21 PM) cbodley: maybe related to configuration from test_assume_role_deny() or test_assume_role_deny_head_nonexistent()?

Actions

Copy link

Updated by Matt Benjamin over 2 years ago

Severity changed from 3 - minor to 2 - major

Actions

Copy link

Updated by Casey Bodley over 2 years ago

Subject changed from rgw: 403 response seen from radosgw on cleanup from (passed!) s3-tests run to rgw/sts: 403 response seen from radosgw on cleanup from (passed!) s3-tests run
Priority changed from High to Urgent

Actions

Copy link

Updated by Casey Bodley over 2 years ago

Assignee set to Kalpesh Pandya
Tags set to sts test

Actions

Copy link

Updated by Casey Bodley over 2 years ago

Hi Kalpesh, it looks like this test is unable to cleanup because of a permission issue. the test either needs to do the cleanup itself with a user that has permission, or change the permissions so that the default user can do the cleanup

Actions

Copy link

Updated by Casey Bodley over 2 years ago

i was looking at a recent run in http://qa-proxy.ceph.com/teuthology/cbodley-2022-01-14_20:20:04-rgw-wip-cbodley-testing-distro-default-smithi/6617281/teuthology.log

022-01-14T20:53:52.155 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_get_session_token ... ok
2022-01-14T20:53:52.183 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_get_session_token_permanent_creds_denied ... ok
2022-01-14T20:53:52.331 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_assume_role_allow ... ok
2022-01-14T20:53:52.407 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_assume_role_deny ... ok
2022-01-14T21:08:52.586 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_assume_role_creds_expiry ... ok
2022-01-14T21:08:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_assume_role_deny_head_nonexistent ... ok
2022-01-14T21:08:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:s3tests_boto3.functional.test_sts.test_assume_role_allow_head_nonexistent ... ok
2022-01-14T21:08:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:ERROR
2022-01-14T21:08:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:
2022-01-14T21:08:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:======================================================================
2022-01-14T21:08:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:ERROR: test suite for <module 's3tests_boto3.functional' from '/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/__init__.py'>
2022-01-14T21:08:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:----------------------------------------------------------------------
2022-01-14T21:08:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:Traceback (most recent call last):
2022-01-14T21:08:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/suite.py", line 229, in run
2022-01-14T21:08:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:    self.tearDown()
2022-01-14T21:08:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/suite.py", line 352, in tearDown
2022-01-14T21:08:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:    self.teardownContext(ancestor)
2022-01-14T21:08:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/suite.py", line 368, in teardownContext
2022-01-14T21:08:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:    try_run(context, names)
2022-01-14T21:08:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
2022-01-14T21:08:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:    return func()
2022-01-14T21:08:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/__init__.py", line 259, in teardown
2022-01-14T21:08:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:    nuke_prefixed_buckets(prefix=prefix, client=alt_client)
2022-01-14T21:08:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/__init__.py", line 148, in nuke_prefixed_buckets
2022-01-14T21:08:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:    buckets = get_buckets_list(client, prefix)
2022-01-14T21:08:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/__init__.py", line 54, in get_buckets_list
2022-01-14T21:08:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:    response = client.list_buckets()
2022-01-14T21:08:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/client.py", line 391, in _api_call
2022-01-14T21:08:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:    return self._make_api_call(operation_name, kwargs)
2022-01-14T21:08:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/client.py", line 719, in _make_api_call
2022-01-14T21:08:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:    raise error_class(parsed_response, operation_name)
2022-01-14T21:08:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Unknown

what sticks out to me is that the failure is under nuke_prefixed_buckets(prefix=prefix, client=alt_client), where `alt_client` is getting AccessDenied from ListBuckets

i see two test cases that attach a "user policy" to this alt_client's user id:
test_get_session_token()
test_get_session_token_permanent_creds_denied()

these user policies deny s3:* actions but allow sts:GetSessionToken. the cleanup code in nuke_prefixed_buckets() would be denied because of this user policy, so those test cases should probably remove that user policy at the end

Actions

Copy link