Project

General

Profile

Bug #44405

Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)

Added by Kefu Chai 11 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Category:
dashboard/rgw
Target version:
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

UPDATE:
https://tracker.ceph.com/issues/44405#note-8

Previous PR:
https://github.com/ceph/ceph/pull/33718

2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:======================================================================
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.735 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/test_rgw.py", line 255, in test_all
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/helper.py", line 365, in assertStatus
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.736 INFO:tasks.cephfs_test_runner:Ran 169 tests in 1769.345s
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:======================================================================
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-03-04T03:16:30.737 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/test_rgw.py", line 255, in test_all
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/mgr/dashboard/helper.py", line 365, in assertStatus
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2020-03-04T03:16:30.738 INFO:tasks.cephfs_test_runner:
2020-03-04T03:16:30.738 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-03-03-1622/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
2020-03-04T03:16:29.197+0000 7f24ca97f700  0 [dashboard DEBUG rest_client] RGW REST API PUT req: /teuth-test-bucket?versioning data: b'<VersioningConfiguration><Status>Suspended</Status><MfaDelete>Disabled</MfaDelete></VersioningConfiguration>'
2020-03-04T03:16:29.197+0000 7f24ca97f700 20 mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2020-03-04T03:16:29.197+0000 7f24ca97f700 10 mgr get_typed_config  REST_REQUESTS_TIMEOUT not found
2020-03-04T03:16:29.197+0000 7f24ca97f700 20 mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2020-03-04T03:16:29.197+0000 7f24ca97f700 10 mgr get_typed_config  REST_REQUESTS_TIMEOUT not found
2020-03-04T03:16:29.201+0000 7f24ca97f700  0 [dashboard DEBUG urllib3.connectionpool] http://172.21.15.139:80 "PUT /teuth-test-bucket?versioning HTTP/1.1" 403 150
2020-03-04T03:16:29.241+0000 7f24ca97f700  0 [dashboard ERROR rest_client] RGW REST API failed PUT req status: 403
2020-03-04T03:16:29.242+0000 7f24ca97f700  0 [dashboard ERROR exception] Dashboard Exception
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 529, in set_bucket_versioning
    request(data=data, headers=headers)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 323, in __call__
    data, raw_content, headers)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 446, in do_request
    resp.content)
dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
(b'{"Code":"AccessDenied","BucketName":"teuth-test-bucket","RequestId":"tx00000'
 b'000000000000002a-005e5f1d8d-11cd-default","HostId":"11cd-default-default"}')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 93, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 665, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 858, in wrapper
    return func(*vpath, **params)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 232, in set
    versioning_state, mfa_delete, mfa_token_serial, mfa_token_pin)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 162, in _set_versioning
    mfa_token_serial, mfa_token_pin)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 521, in func_wrapper
    **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 538, in set_bucket_versioning
    component='rgw')
dashboard.exceptions.DashboardException: Bad MFA credentials: RGW REST API failed request with status code 403
(b'{"Code":"AccessDenied","BucketName":"teuth-test-bucket","RequestId":"tx00000'
 b'000000000000002a-005e5f1d8d-11cd-default","HostId":"11cd-default-default"}')
2020-03-04T03:16:29.243+0000 7f24ca97f700  0 [dashboard INFO request] [::ffff:172.21.0.51:47928] [PUT] [400] [0.062s] [admin] [221.0B] /api/rgw/bucket/teuth-test-bucket

Related issues

Related to mgr - Bug #46735: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) New
Copied to mgr - Backport #45233: octopus: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) Resolved

History

#1 Updated by Kefu Chai 11 months ago

Alfonso, could you help take a look?

see http://pulpito.ceph.com/kchai-2020-03-03_13:02:55-rados-wip-kefu-testing-2020-03-03-1622-distro-basic-smithi/4820571/

it was a test branch, but it didn't include any rgw/mgr/dashboard related changes. so i am wondering if this is an issue which also exists in master.

#2 Updated by Alfonso Martínez 11 months ago

Sure, Kefu.

Looking at radosgw log, particularly civetweb timestamps:

2020-03-04T03:16:28.086+0000 7fc34f90d700  1 ====== req done req=0x7fc34f904ed0 op status=0 http_status=200 latency=0.003999911s ======
2020-03-04T03:16:28.086+0000 7fc34f90d700  1 civetweb: 0x55a31bcae9d8: 172.21.15.139 - - [04/Mar/2020:03:16:27 +0000] "PUT /teuth-test-bucket?versioning HTTP/1.1" 200 197 - python-requests/2.20.0
[...]
2020-03-04T03:16:29.200+0000 7fc34f90d700  1 ====== req done req=0x7fc34f904ed0 op status=-2044 http_status=403 latency=0.001999955s ======
2020-03-04T03:16:29.200+0000 7fc34f90d700  1 civetweb: 0x55a31bcae9d8: 172.21.15.139 - - [04/Mar/2020:03:16:29 +0000] "PUT /teuth-test-bucket?versioning HTTP/1.1" 403 379 - python-requests/2.20.0

It seems to me that due to the time gap between requests that include
MFA TOTP token, it is needed to increase the time between token generation ([[https://docs.ceph.com/docs/master/radosgw/mfa/]]).

In ceph-master-nightly jenkins job this is not happening.

#3 Updated by Alfonso Martínez 11 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 33718

#4 Updated by Sage Weil 11 months ago

  • Status changed from Fix Under Review to Resolved

#5 Updated by Laura Paduano 10 months ago

  • Copied to Backport #44890: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added

#7 Updated by Brad Hubbard 9 months ago

  • Status changed from Resolved to New

#8 Updated by Alfonso Martínez 9 months ago

I ran this tests several times locally and only once I faced the same issue, so
I guess that depending on requests' timestamps falling into the time window you can face
this issue.
In this PR the time between TOTP retrievals is increased to ensure that the time window for 1st validated TOTP is expired:
https://github.com/ceph/ceph/pull/34666

If this issue appears again, we should investigate it with RGW team.

#9 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)

#10 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)
  • Status changed from New to Resolved

#11 Updated by Alfonso Martínez 9 months ago

  • Backport set to octopus

#12 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)
  • Pull request ID changed from 33718 to 34666

#13 Updated by Alfonso Martínez 9 months ago

  • Status changed from Resolved to Pending Backport

#14 Updated by Alfonso Martínez 9 months ago

  • Copied to deleted (Backport #44890: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest))

#15 Updated by Alfonso Martínez 9 months ago

  • Copied to Backport #45233: octopus: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added

#16 Updated by Lenz Grimmer 9 months ago

  • Status changed from Pending Backport to Resolved
  • Target version set to v16.0.0

#17 Updated by Sebastian Wagner 6 months ago

  • Related to Bug #46735: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added

Also available in: Atom PDF