Bug #44405: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) - Dashboard - Ceph

Actions

Copy link

Bug #44405

closed

Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)

Added by Kefu Chai about 4 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Alfonso Martínez

Category:

Component - RGW

Target version:

Ceph - v16.0.0

% Done:

Source:

Tags:

Backport:

octopus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

34666

Crash signature (v1):

Crash signature (v2):

Description

UPDATE:
https://tracker.ceph.com/issues/44405#note-8

Previous PR:
https://github.com/ceph/ceph/pull/33718

2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:======================================================================
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/test_rgw.py", line 255, in test_all
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/helper.py", line 365, in assertStatus
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:Ran 169 tests in 1769.345s
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:======================================================================
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/test_rgw.py", line 255, in test_all
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/helper.py", line 365, in assertStatus
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.738 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))

2020-03-04T03:16:29.197+0000 7f24ca97f700  0 [dashboard DEBUG rest_client] RGW REST API PUT req: /teuth-test-bucket?versioning data: b'<VersioningConfiguration><Status>Suspended</Status><MfaDelete>Disabled</MfaDelete></VersioningConfiguration>'
2020-03-04T03:16:29.197+0000 7f24ca97f700 20 mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2020-03-04T03:16:29.197+0000 7f24ca97f700 10 mgr get_typed_config  REST_REQUESTS_TIMEOUT not found
2020-03-04T03:16:29.197+0000 7f24ca97f700 20 mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2020-03-04T03:16:29.197+0000 7f24ca97f700 10 mgr get_typed_config  REST_REQUESTS_TIMEOUT not found
2020-03-04T03:16:29.201+0000 7f24ca97f700  0 [dashboard DEBUG urllib3.connectionpool] http://172.21.15.139:80 "PUT /teuth-test-bucket?versioning HTTP/1.1" 403 150
2020-03-04T03:16:29.241+0000 7f24ca97f700  0 [dashboard ERROR rest_client] RGW REST API failed PUT req status: 403
2020-03-04T03:16:29.242+0000 7f24ca97f700  0 [dashboard ERROR exception] Dashboard Exception
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 529, in set_bucket_versioning
    request(data=data, headers=headers)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 323, in __call__
    data, raw_content, headers)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 446, in do_request
    resp.content)
dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
(b'{"Code":"AccessDenied","BucketName":"teuth-test-bucket","RequestId":"tx00000'
 b'000000000000002a-005e5f1d8d-11cd-default","HostId":"11cd-default-default"}')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 93, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 665, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 858, in wrapper
    return func(*vpath, **params)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 232, in set
    versioning_state, mfa_delete, mfa_token_serial, mfa_token_pin)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 162, in _set_versioning
    mfa_token_serial, mfa_token_pin)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 521, in func_wrapper
    **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 538, in set_bucket_versioning
    component='rgw')
dashboard.exceptions.DashboardException: Bad MFA credentials: RGW REST API failed request with status code 403
(b'{"Code":"AccessDenied","BucketName":"teuth-test-bucket","RequestId":"tx00000'
 b'000000000000002a-005e5f1d8d-11cd-default","HostId":"11cd-default-default"}')
2020-03-04T03:16:29.243+0000 7f24ca97f700  0 [dashboard INFO request] [::ffff:172.21.0.51:47928] [PUT] [400] [0.062s] [admin] [221.0B] /api/rgw/bucket/teuth-test-bucket

Related issues 2 (1 open — 1 closed)

Actions

Copy link

Updated by Kefu Chai about 4 years ago

Alfonso, could you help take a look?

see http://pulpito.ceph.com/kchai-2020-03-03_13:02:55-rados-wip-kefu-testing-2020-03-03-1622-distro-basic-smithi/4820571/

it was a test branch, but it didn't include any rgw/mgr/dashboard related changes. so i am wondering if this is an issue which also exists in master.

Actions

Copy link

Updated by Alfonso Martínez about 4 years ago

Sure, Kefu.

Looking at radosgw log, particularly civetweb timestamps:

2020-03-04T03:16:28.086+0000 7fc34f90d700  1 ====== req done req=0x7fc34f904ed0 op status=0 http_status=200 latency=0.003999911s ======
2020-03-04T03:16:28.086+0000 7fc34f90d700  1 civetweb: 0x55a31bcae9d8: 172.21.15.139 - - [04/Mar/2020:03:16:27 +0000] "PUT /teuth-test-bucket?versioning HTTP/1.1" 200 197 - python-requests/2.20.0
[...]
2020-03-04T03:16:29.200+0000 7fc34f90d700  1 ====== req done req=0x7fc34f904ed0 op status=-2044 http_status=403 latency=0.001999955s ======
2020-03-04T03:16:29.200+0000 7fc34f90d700  1 civetweb: 0x55a31bcae9d8: 172.21.15.139 - - [04/Mar/2020:03:16:29 +0000] "PUT /teuth-test-bucket?versioning HTTP/1.1" 403 379 - python-requests/2.20.0

It seems to me that due to the time gap between requests that include
MFA TOTP token, it is needed to increase the time between token generation ([[https://docs.ceph.com/docs/master/radosgw/mfa/]]).

In ceph-master-nightly jenkins job this is not happening.

Actions

Copy link

Updated by Alfonso Martínez about 4 years ago

Status changed from New to Fix Under Review
Pull request ID set to 33718

Actions

Copy link

Updated by Sage Weil about 4 years ago

Status changed from Fix Under Review to Resolved

Actions

Copy link

Updated by Laura Paduano about 4 years ago

Copied to Backport #44890: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

Seeing this in a branch that includes PR 33718

http://pulpito.ceph.com/bhubbard-2020-04-16_09:57:54-rados-wip-badone-testing-distro-basic-smithi/4958030/

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

Status changed from Resolved to New

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

I ran this tests several times locally and only once I faced the same issue, so
I guess that depending on requests' timestamps falling into the time window you can face
this issue.
In this PR the time between TOTP retrievals is increased to ensure that the time window for 1st validated TOTP is expired:
https://github.com/ceph/ceph/pull/34666

If this issue appears again, we should investigate it with RGW team.

Actions

Copy link