Actions
Bug #58059
closeds3tests v2 SignatureDoesNotMatch failures on ubuntu
Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
from main branch results: https://pulpito.ceph.com/cbodley-2022-11-21_18:00:47-rgw-main-distro-default-smithi/
s3tests passed on centos jobs, but all ubuntu jobs failed
ERROR: s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the CreateBucket operation: Unknown ERROR: s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the CreateBucket operation: Unknown ERROR: s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutBucketPolicy operation: Unknown ERROR: s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutBucketPolicy operation: Unknown
Updated by Casey Bodley over 1 year ago
boto versions from a failing run on ubuntu:
2022-11-21T18:59:47.169 INFO:teuthology.orchestra.run.smithi055.stdout:Collecting boto>=2.6.0 2022-11-21T18:59:47.195 INFO:teuthology.orchestra.run.smithi055.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2022-11-21T18:59:47.284 INFO:teuthology.orchestra.run.smithi055.stdout: ---------------------------------------- 1.4/1.4 MB 16.1 MB/s eta 0:00:00 2022-11-21T18:59:47.833 INFO:teuthology.orchestra.run.smithi055.stdout:Collecting boto3>=1.0.0 2022-11-21T18:59:47.862 INFO:teuthology.orchestra.run.smithi055.stdout: Downloading boto3-1.26.13-py3-none-any.whl (132 kB) 2022-11-21T18:59:49.955 INFO:teuthology.orchestra.run.smithi055.stdout:Collecting botocore<1.30.0,>=1.29.13 2022-11-21T18:59:49.975 INFO:teuthology.orchestra.run.smithi055.stdout: Downloading botocore-1.29.13-py3-none-any.whl (9.9 MB) 2022-11-21T18:59:50.181 INFO:teuthology.orchestra.run.smithi055.stdout: ---------------------------------------- 9.9/9.9 MB 49.4 MB/s eta 0:00:00
from a successful run on centos:
2022-11-21T19:04:14.146 INFO:teuthology.orchestra.run.smithi137.stdout:Collecting boto>=2.6.0 2022-11-21T19:04:14.150 INFO:teuthology.orchestra.run.smithi137.stdout: Using cached boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2022-11-21T19:04:14.945 INFO:teuthology.orchestra.run.smithi137.stdout:Collecting boto3>=1.0.0 2022-11-21T19:04:14.947 INFO:teuthology.orchestra.run.smithi137.stdout: Using cached boto3-1.23.10-py3-none-any.whl (132 kB) 2022-11-21T19:04:17.814 INFO:teuthology.orchestra.run.smithi137.stdout:Collecting botocore<1.27.0,>=1.26.10 2022-11-21T19:04:17.828 INFO:teuthology.orchestra.run.smithi137.stdout: Using cached botocore-1.26.10-py3-none-any.whl (8.8 MB)
Updated by Casey Bodley over 1 year ago
- Subject changed from s3tests SignatureDoesNotMatch failures on ubuntu to s3tests v2 SignatureDoesNotMatch failures on ubuntu
bisected botocore versions down to good=botocore-1.27.96 bad=botocore-1.28.0
botocore debug log output from good=botocore-1.27.96:
botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/yournamehere-vosguleoy6r2ngj1-1', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': ''}, 'body': b'', 'url': 'http://localhost:8000/yournamehere-vosguleoy6r2ngj1-1', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fc207fe6fa0>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'yournamehere-vosguleoy6r2ngj1-1'}}}
botocore.hooks [DEBUG] Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc207fe6f10>>
botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fc207fe6070>>
botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x7fc2099101f0>
botocore.hooks [DEBUG] Event before-sign.s3.CreateBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fc207fe6070>>
botocore.utils [DEBUG] Using S3 path style addressing.
botocore.auth [DEBUG] Calculating signature using hmacv1 auth.
botocore.auth [DEBUG] HTTP request method: PUT
botocore.auth [DEBUG] StringToSign:
PUT
Tue, 22 Nov 2022 16:54:57 GMT
/yournamehere-vosguleoy6r2ngj1-1
same test case output from bad=botocore-1.28.0:
botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': ''}, 'body': b'', 'auth_path': '/yournamehere-cn8gsp4kpyz4mhoo-1/', 'url': 'http://localhost:8000/yournamehere-cn8gsp4kpyz4mhoo-1', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f24a7ccb370>, 'has_streaming_input': False, 'auth_type': None, 'signing': {}, 's3_redirect': {'redirected': False, 'bucket': 'yournamehere-cn8gsp4kpyz4mhoo-1', 'params': {'Bucket': 'yournamehere-cn8gsp4kpyz4mhoo-1'}}}}
botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f24a7ccb2e0>>
botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x7f24a9741550>
botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler <function remove_arn_from_signing_path at 0x7f24a9746790>
botocore.auth: DEBUG: Calculating signature using hmacv1 auth.
botocore.auth: DEBUG: HTTP request method: PUT
botocore.auth: DEBUG: StringToSign:
PUT
Tue, 22 Nov 2022 16:51:20 GMT
/yournamehere-cn8gsp4kpyz4mhoo-1/
the bad version adds a trailing slash to the request path in the StringToSign, which corresponds to the new 'auth_path' parameter in the OperationModel
Updated by Casey Bodley over 1 year ago
- Status changed from New to Fix Under Review
Updated by Casey Bodley over 1 year ago
- Status changed from Fix Under Review to Resolved
Updated by Casey Bodley about 2 months ago
- Related to Bug #64801: s3tests: unpin botocore version added
Actions