Project

General

Profile

Bug #36034

multisite: intermittent test_bucket_index_log_trim failures

Added by Casey Bodley 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
09/17/2018
Due date:
% Done:

0%

Source:
Tags:
multisite
Backport:
mimic luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

The 'bilog autotrim' command in this test sometimes fails to trim all entries, causing a failed assert(len(active_bilog) == 0).

From http://qa-proxy.ceph.com/teuthology/cbodley-2018-09-17_14:26:55-rgw:multisite-wip-multisite-testing-distro-basic-smithi/3033357/teuthology.log:

2018-09-17T15:05:58.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0']
2018-09-17T15:05:58.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0']
2018-09-17T15:05:58.143 INFO:teuthology.orchestra.run.smithi037:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0'
2018-09-17T15:05:59.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'soltxr-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false']
2018-09-17T15:05:59.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'soltxr-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false']
2018-09-17T15:05:59.809 INFO:teuthology.orchestra.run.smithi037:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket soltxr-46 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false'
2018-09-17T15:06:00.624 INFO:teuthology.orchestra.run.smithi037.stdout:[{"op_id":"00000000001.31.4","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1351","op":"write","object":"a","instance":"","state":"pending","index_ver":1,"timestamp":"0.000000","ver":{"pool":-1,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000002.32.5","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1351","op":"write","object":"a","instance":"","state":"complete","index_ver":2,"timestamp":"2018-09-17 15:05:47.328407316Z","ver":{"pool":8,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000003.33.4","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1353","op":"write","object":"b","instance":"","state":"pending","index_ver":3,"timestamp":"0.000000","ver":{"pool":-1,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000004.34.5","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1353","op":"write","object":"b","instance":"","state":"complete","index_ver":4,"timestamp":"2018-09-17 15:05:47.502896761Z","ver":{"pool":8,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000005.35.4","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1355","op":"write","object":"c","instance":"","state":"pending","index_ver":5,"timestamp":"0.000000","ver":{"pool":-1,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000006.36.5","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1355","op":"write","object":"c","instance":"","state":"complete","index_ver":6,"timestamp":"2018-09-17 15:05:47.657797139Z","ver":{"pool":8,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000007.37.4","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1358","op":"write","object":"d","instance":"","state":"pending","index_ver":7,"timestamp":"0.000000","ver":{"pool":-1,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]},{"op_id":"00000000008.38.5","op_tag":"6b494045-35f4-4a6d-9a86-dea6396200e2.4559.1358","op":"write","object":"d","instance":"","state":"complete","index_ver":8,"timestamp":"2018-09-17 15:05:47.848360747Z","ver":{"pool":8,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":["6b494045-35f4-4a6d-9a86-dea6396200e2"]}]
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:======================================================================
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2018-09-17T15:06:00.635 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-multisite-testing/qa/tasks/rgw_multi/tests.py", line 1120, in test_bucket_index_log_trim
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:    assert(len(active_bilog) == 0)
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:AssertionError
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:Ran 21 tests in 787.673s
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:
2018-09-17T15:06:00.636 INFO:tasks.rgw_multisite_tests:FAILED (failures=1)


Related issues

Copied to rgw - Backport #36201: mimic: multisite: intermittent test_bucket_index_log_trim failures Resolved
Copied to rgw - Backport #36202: luminous: multisite: intermittent test_bucket_index_log_trim failures Resolved

History

#1 Updated by Casey Bodley 3 months ago

  • Status changed from New to Need Review
  • Backport set to mimic luminous

#2 Updated by Casey Bodley 3 months ago

  • Status changed from Need Review to Pending Backport

#3 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #36201: mimic: multisite: intermittent test_bucket_index_log_trim failures added

#4 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #36202: luminous: multisite: intermittent test_bucket_index_log_trim failures added

#5 Updated by Nathan Cutler about 2 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF