Project

General

Profile

Actions
Copy link

Bug #63486

open

reef: test_lifecycle_cloud_transition_large_obj FAILED

Added by Casey Bodley 6 months ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Soumya Koduri
Target version:
-
% Done:

0%

Source:
Tags:
cloud-transition
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v18.2.1
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-07_20:50:58-rgw-reef-release-distro-default-smithi/7451553/teuthology.log

2023-11-07T22:02:33.509 INFO:teuthology.orchestra.run.smithi039.stdout:=================================== FAILURES ===================================
2023-11-07T22:02:33.509 INFO:teuthology.orchestra.run.smithi039.stdout:__________________ test_lifecycle_cloud_transition_large_obj ___________________
...
2023-11-07T22:02:33.516 INFO:teuthology.orchestra.run.smithi039.stdout:        # Wait for first expiration (plus fudge to handle the timer window)
2023-11-07T22:02:33.516 INFO:teuthology.orchestra.run.smithi039.stdout:        time.sleep(8*lc_interval)
2023-11-07T22:02:33.516 INFO:teuthology.orchestra.run.smithi039.stdout:        expire1_keys = list_bucket_storage_class(client, bucket)
2023-11-07T22:02:33.517 INFO:teuthology.orchestra.run.smithi039.stdout:>       assert len(expire1_keys['STANDARD']) == 1
2023-11-07T22:02:33.517 INFO:teuthology.orchestra.run.smithi039.stdout:E       assert 2 == 1
2023-11-07T22:02:33.517 INFO:teuthology.orchestra.run.smithi039.stdout:E        +  where 2 = len([{'ETag': '"571238d94d855de986ff1ac818941dcc"', 'IsLatest': True, 'Key': 'expire1/multi', 'LastModified': datetime.datetime(2023, 11, 7, 22, 1, 12, 938000, tzinfo=tzlocal()), ...}, {'ETag': '"571238d94d855de986ff1ac818941dcc"', 'IsLatest': True, 'Key': 'keep/multi', 'LastModified': datetime.datetime(2023, 11, 7, 22, 1, 12, 811000, tzinfo=tzlocal()), ...}])
2023-11-07T22:02:33.517 INFO:teuthology.orchestra.run.smithi039.stdout:
2023-11-07T22:02:33.517 INFO:teuthology.orchestra.run.smithi039.stdout:s3tests_boto3/functional/test_s3.py:9092: AssertionError
2023-11-07T22:02:33.518 INFO:teuthology.orchestra.run.smithi039.stdout:--------------------------- Captured stdout teardown
Actions
Copy link
 #1

Updated by Casey Bodley about 1 month ago

again on reef, a failure from test_lifecycle_noncur_cloud_transition in https://qa-proxy.ceph.com/teuthology/cbodley-2024-03-26_12:30:03-rgw-wip-63856-reef-distro-default-smithi/7623190/teuthology.log

2024-03-26T15:50:31.546 INFO:teuthology.orchestra.run.smithi080.stdout:=================================== FAILURES ===================================
2024-03-26T15:50:31.546 INFO:teuthology.orchestra.run.smithi080.stdout:____________________ test_lifecycle_noncur_cloud_transition ____________________
2024-03-26T15:50:31.546 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.lifecycle
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.lifecycle_expiration
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.lifecycle_transition
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.cloud_transition
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.fails_on_aws
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    @pytest.mark.fails_on_dbstore
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:    def test_lifecycle_noncur_cloud_transition():
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        cloud_sc = get_cloud_storage_class()
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        if cloud_sc == None:
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:            pytest.skip('[s3 cloud] section missing cloud_storage_class')
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        retain_head_object = get_cloud_retain_head_object()
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        target_path = get_cloud_target_path()
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        target_sc = get_cloud_target_storage_class()
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.547 INFO:teuthology.orchestra.run.smithi080.stdout:        sc1 = get_cloud_regular_storage_class()
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        if (sc1 == None):
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:            pytest.skip('[s3 cloud] section missing storage_class')
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        sc = ['STANDARD', sc1, cloud_sc]
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        bucket = get_new_bucket()
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        client = get_client()
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        check_configure_versioning_retry(bucket, "Enabled", "Enabled")
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:        rules = [
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:            {
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:                'ID': 'rule1',
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:                'Prefix': 'test1/',
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:                'Status': 'Enabled',
2024-03-26T15:50:31.548 INFO:teuthology.orchestra.run.smithi080.stdout:                'NoncurrentVersionTransitions': [
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                    {
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                        'NoncurrentDays': 1,
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                        'StorageClass': sc[1]
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                    },
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                    {
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                        'NoncurrentDays': 5,
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                        'StorageClass': sc[2]
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                    }
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:                ],
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:            }
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:        ]
2024-03-26T15:50:31.549 INFO:teuthology.orchestra.run.smithi080.stdout:        lifecycle = {'Rules': rules}
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        response = client.put_bucket_lifecycle_configuration(Bucket=bucket, LifecycleConfiguration=lifecycle)
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        keys = ['test1/a', 'test1/b']
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        for k in keys:
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:            create_multiple_versions(client, bucket, k, 3)
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        init_keys = list_bucket_storage_class(client, bucket)
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        assert len(init_keys['STANDARD']) == 6
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:        response  = client.list_object_versions(Bucket=bucket)
2024-03-26T15:50:31.550 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        lc_interval = get_lc_debug_interval()
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        time.sleep(4*lc_interval)
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        expire1_keys = list_bucket_storage_class(client, bucket)
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        assert len(expire1_keys['STANDARD']) == 2
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        assert len(expire1_keys[sc[1]]) == 4
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        assert len(expire1_keys[sc[2]]) == 0
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        time.sleep(10*lc_interval)
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        expire1_keys = list_bucket_storage_class(client, bucket)
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:        assert len(expire1_keys['STANDARD']) == 2
2024-03-26T15:50:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:>       assert len(expire1_keys[sc[1]]) == 0
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:E       assert 4 == 0
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:E         +4
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:E         -0
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:s3tests_boto3/functional/test_s3.py:9036: AssertionError
2024-03-26T15:50:31.552 INFO:teuthology.orchestra.run.smithi080.stdout:--------------------------- Captured stdout teardown ---------------------------
Actions
Copy link
 #2

Updated by Casey Bodley about 1 month ago

  • Assignee set to Soumya Koduri
Actions
Copy link
 #3

Updated by Soumya Koduri about 1 month ago

  • Status changed from New to In Progress

@Casey Bodley,

Any idea if these failures are specific to only reef branch?

Actions
Copy link
 #4

Updated by Casey Bodley about 1 month ago

Soumya Koduri wrote:

@Casey Bodley,

Any idea if these failures are specific to only reef branch?

i think i've only seen that on reef

Actions
Copy link

Also available in: Atom PDF