Project

General

Profile

Actions
Copy link

Bug #43661

closed

failure in test_sse_kms_post_object_authenticated_request

Added by Kefu Chai over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Ali Maredia
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
32940
Crash signature (v1):
Crash signature (v2):

Description

2020-01-19T08:53:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:======================================================================
2020-01-19T08:53:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:FAIL: s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request
2020-01-19T08:53:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------------------------
2020-01-19T08:53:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last):
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:    self.test(*self.arg)
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 10235, in test_sse_kms_post_object_authenticated_re
quest
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:    eq(r.status_code, 204)
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:AssertionError: 400 != 204

2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:-------------------- >> begin captured logging << --------------------
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f41b2ba9bf8>
2020-01-19T08:53:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f41b2c162f0>
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f41b248ed08>
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f41b2c160d0>
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler <function validate_bucket_name at 0x7f41b2745048>
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f41b1b8f9b0>>
2020-01-19T08:53:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f41b1b8fda0>>
2020-01-19T08:53:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler <function generate_idempotent_uuid at 0x7f41b2743bf8>
2020-01-19T08:53:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler <function add_expect_header at 0x7f41b2745378>
2020-01-19T08:53:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f41b1b8f9b0>>
2020-01-19T08:53:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler <function inject_api_version_header_if_needed at 0x7f41b2747620>
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-awpzsyl6p41dn7b-389', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.11.5 Python/3.6.8 Linux/4.18.0-147.3.1.el8_1.x86_64 Botocore/1.14.5'}, 'body': b'', 'url': 'http://smithi067.front.sepia.ceph.com:80/test-client.0-awpzsyl6p41dn7b-389', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f41b1b8f668>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-awpzsyl6p41dn7b-389'}}}
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f41b1b8f5f8>>
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x7f41b2743ae8>
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f41b1b8fe10>>
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.utils: DEBUG: Using S3 path style addressing.
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth.
2020-01-19T08:53:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.auth: DEBUG: CanonicalRequest:
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:PUT
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:/test-client.0-awpzsyl6p41dn7b-389
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:host:smithi067.front.sepia.ceph.com
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-01-19T08:53:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:x-amz-date:20200119T085133Z
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:host;x-amz-content-sha256;x-amz-date
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.auth: DEBUG: StringToSign:
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:AWS4-HMAC-SHA256
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:20200119T085133Z
2020-01-19T08:53:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:20200119/us-east-1/s3/aws4_request
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:19cf9d9b5231ffce0caf411304de31e40e3cc9a2fbd55f7a9a738be2723c0185
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.auth: DEBUG: Signature:
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:214f144e77dbca54d2df5d654a571f0db356c0a3e8cc9d24d090c3d7b7948f71
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.endpoint: DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://smithi067.front.sepia.ceph.com:80/test-client.0-awpzsyl6p41dn7b-389, headers={'User-Agent': b'Boto3/1.11.5 Python/3.6.8 Linux/4.18.0-147.3.1.el8_1.x86_64 Botocore/1.14.5', 'X-Amz-Date': b'20200119T085133Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=MGOZSUMOWLYTGAIUXUWV/20200119/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=214f144e77dbca54d2df5d654a571f0db356c0a3e8cc9d24d090c3d7b7948f71', 'Content-Length': '0'}>
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi067.front.sepia.ceph.com:80
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:urllib3.connectionpool: DEBUG: http://smithi067.front.sepia.ceph.com:80 "PUT /test-client.0-awpzsyl6p41dn7b-389 HTTP/1.1" 200 0
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000000000000000001135-005e241895-1146-default', 'Content-Length': '0', 'Date': 'Sun, 19 Jan 2020 08:51:33 GMT'}
2020-01-19T08:53:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.parsers: DEBUG: Response body:
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:b''
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7f41b1b8f9e8>
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.retryhandler: DEBUG: No retry needed.
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f41b1b8f9b0>>
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f41b2ba9bf8>
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f41b2c162f0>
2020-01-19T08:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f41b248ed08>
2020-01-19T08:53:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f41b2c160d0>
2020-01-19T08:53:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
2020-01-19T08:53:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3
2020-01-19T08:53:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi067.front.sepia.ceph.com:80
2020-01-19T08:53:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:urllib3.connectionpool: DEBUG: http://smithi067.front.sepia.ceph.com:80 "POST /test-client.0-awpzsyl6p41dn7b-389 HTTP/1.1" 400 245

/a//kchai-2020-01-19_03:33:01-rados-wip-kefu-testing-2020-01-18-2208-distro-basic-smithi/4682624/

Related issues 2 (0 open — 2 closed)

Related to rgw - Bug #41212: Failure in test_encryption_sse_c_post_object_authenticated_request and test_sse_kms_post_object_authenticated_requestResolvedCasey Bodley08/12/2019

Actions
Has duplicate rgw - Bug #43842: s3-tests failure in rados suiteDuplicate

Actions
Actions
Copy link
 #1

Updated by Kefu Chai over 4 years ago

  • Related to Bug #41212: Failure in test_encryption_sse_c_post_object_authenticated_request and test_sse_kms_post_object_authenticated_request added
Actions
Copy link
 #2

Updated by Casey Bodley over 4 years ago

  • Status changed from New to Triaged

some of the s3test cases rely on special configuration, which the rgw suite provides in ex. qa/suites/rgw/verify/overrides.yaml:

overrides:
  ceph:
    conf:
      client:
        rgw crypt s3 kms backend: testing
        rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=
        rgw crypt require ssl: false

We'll need to apply those to any suites running the s3test task, so we should probably put that in a common file like qa/rgw/overrides/s3tests.yaml and symlink it into the suites that need it.

But it's not clear to me that the rados suite should be running s3tests at all - we made this change after removing s3 readwrite and roundtrip tests because we don't see much value in maintaining them. Do you folks on the rados team think it makes sense to run rgw tests in your suite?

Actions
Copy link
 #3

Updated by Kefu Chai over 4 years ago

  • Subject changed from failure in test_sse_kms_post_object_authenticated_re quest to failure in test_sse_kms_post_object_authenticated_request
Actions
Copy link
 #4

Updated by Kefu Chai over 4 years ago

just a little more context regarding this test in rados: it was introduced by https://github.com/ceph/ceph-qa-suite/pull/451 for some real world workload driven by rgw. but we also added some synthesized test driven by librados's unittest. see https://github.com/ceph/ceph/pull/4803, which is in turn the fix for #11677

Actions
Copy link
 #5

Updated by Kefu Chai over 4 years ago

[07:53:24] <sjust_> kefu: Is there something problematic with that test combination?
[07:53:34] <sjust_> It's merely running s3tests while doing pool snapshots
[07:54:05] <kefu> sjust_ no. we just need to tweak the setting a little bit to cater the needs of the recent change in s3-tests.
[07:54:14] <sjust_> Ok, that sounds fine then.
[07:54:21] <kefu> but since we already have the tests in rgw test suite
[07:54:34] <kefu> Casey wonder if we really need do this in rados suite also.
[07:54:35] <sjust_> I think it got added because at some point rgw broke when combined with pool snaps -- this was a way to detect that
[07:54:43] <sjust_> I don't really have an opinion about whether we keep it around
[07:54:46] <kefu> yes, that's also my understanding.
[07:55:07] <sjust_> You could also just move that test combo to rgw
[07:55:29] <sjust_> Or get rid of it -- I'm not sure anyone really uses pool snapshots with rgw

Casey, if nobody is really using pool snapshots with rgw. we can just drop this test. otherwise, i think we can add a thash workload to thrash_pool_snaps when exercising s3tests to test that combination.

Actions
Copy link
 #6

Updated by Casey Bodley about 4 years ago

  • Has duplicate Bug #43842: s3-tests failure in rados suite added
Actions
Copy link
 #7

Updated by Casey Bodley about 4 years ago

Thanks Sam and Kefu - since radosgw uses several different pools, a consistent snapshot would require snapshotting each of those pools while no mutations were taking place. Since we don't have any documentation describing this process, I'd say that we just don't support snapshots. So I agree that we should remove this functional test coverage from the rados suite.

Actions
Copy link
 #8

Updated by Ali Maredia about 4 years ago

  • Assignee set to Ali Maredia
  • Pull request ID set to 32940

I have issued a pull request to remove the s3-tests from the rados test suite.

Actions
Copy link
 #9

Updated by Sage Weil about 4 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #10

Updated by Sage Weil about 4 years ago

  • Status changed from Triaged to Fix Under Review
Actions
Copy link
 #11

Updated by Sage Weil about 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF