Bug #58917
Updated by Casey Bodley about 1 year ago
seems to reproduce more in the realms/three-zones.yaml configuration
ex. http://qa-proxy.ceph.com/teuthology/cbodley-2023-03-03_13:50:55-rgw-wip-cbodley-testing-distro-default-smithi/7192301/teuthology.log
ex. http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_18:23:18-rgw-reef-distro-default-smithi/7193368/teuthology.log
<pre>
2023-03-03T16:55:51.950 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:55:51.951 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync
2023-03-03T16:55:51.951 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:55:51.953 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:55:51.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:55:51.954 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:55:51.954 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 921, in test_multi_period_incremental_sync
2023-03-03T16:55:51.954 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(z1)
2023-03-03T16:55:51.954 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:55:51.955 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:55:51.955 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:55:51.955 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:55:54.069 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:55:54.069 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect
2023-03-03T16:55:54.069 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:55:54.071 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:55:54.071 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:55:54.071 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:55:54.072 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1028, in test_multi_zone_redirect
2023-03-03T16:55:54.072 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:55:54.072 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:55:54.072 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:55:54.073 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:55:54.073 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:55:54.073 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:55:54.073 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:55:56.170 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:55:56.170 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove
2023-03-03T16:55:56.170 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:55:56.172 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:55:56.172 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:55:56.172 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:55:56.173 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1057, in test_zonegroup_remove
2023-03-03T16:55:56.173 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:55:56.173 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:55:56.173 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:55:56.174 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:55:56.174 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:55:56.174 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:55:56.174 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:55:58.265 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:55:58.265 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync
2023-03-03T16:55:58.265 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:55:58.267 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:55:58.267 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:55:58.268 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:55:58.268 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1280, in test_encrypted_object_sync
2023-03-03T16:55:58.268 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:55:58.268 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:55:58.269 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:55:58.269 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:55:58.269 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:55:58.269 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:55:58.269 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:00.271 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:00.271 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim
2023-03-03T16:56:00.271 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:00.273 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:00.273 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:00.274 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:00.274 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1310, in test_bucket_index_log_trim
2023-03-03T16:56:00.274 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket()
2023-03-03T16:56:00.274 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1305, in make_test_bucket
2023-03-03T16:56:00.274 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:00.275 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:00.275 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:00.275 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:00.275 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:00.276 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:00.276 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:02.310 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:02.310 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim
2023-03-03T16:56:02.310 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:02.312 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:02.312 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:02.312 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:02.313 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1364, in test_bucket_reshard_index_log_trim
2023-03-03T16:56:02.313 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket()
2023-03-03T16:56:02.313 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1359, in make_test_bucket
2023-03-03T16:56:02.313 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:02.314 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:02.314 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:02.314 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:02.314 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:02.314 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:02.315 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:04.445 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:04.446 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental
2023-03-03T16:56:04.446 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:04.447 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:04.447 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:04.447 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:04.448 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1441, in test_bucket_reshard_incremental
2023-03-03T16:56:04.448 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:04.448 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:04.448 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:04.449 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:04.449 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:04.449 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:04.449 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:06.390 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:06.390 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full
2023-03-03T16:56:06.390 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:06.392 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:06.392 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:06.392 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:06.393 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1471, in test_bucket_reshard_full
2023-03-03T16:56:06.393 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:06.393 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:06.393 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:06.393 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:06.394 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:06.394 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:06.394 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:08.330 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:08.330 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time
2023-03-03T16:56:08.330 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:08.332 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:08.332 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:08.332 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:08.333 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1504, in test_bucket_creation_time
2023-03-03T16:56:08.333 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:08.333 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:08.333 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:08.334 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:08.334 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:08.334 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:08.334 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:14.176 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:14.176 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init
2023-03-03T16:56:14.177 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:14.178 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:14.178 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:14.179 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:14.179 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1717, in test_bucket_full_sync_after_data_sync_init
2023-03-03T16:56:14.179 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:14.179 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:14.180 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:14.180 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:14.180 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:14.180 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:14.180 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:16.135 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:16.136 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init
2023-03-03T16:56:16.136 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:16.137 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:16.138 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:16.138 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:16.138 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1749, in test_resharded_bucket_full_sync_after_data_sync_init
2023-03-03T16:56:16.138 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:16.139 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:16.139 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:16.139 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:16.139 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:16.140 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:16.140 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:18.081 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:18.081 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init
2023-03-03T16:56:18.081 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:18.083 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:18.083 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:18.083 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:18.083 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1788, in test_bucket_incremental_sync_after_data_sync_init
2023-03-03T16:56:18.084 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:18.084 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:18.084 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:18.084 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:18.085 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:18.085 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:18.085 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:20.089 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:20.089 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init
2023-03-03T16:56:20.089 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:20.091 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:20.091 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:20.091 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:20.092 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1826, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init
2023-03-03T16:56:20.092 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:20.092 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:20.092 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:20.093 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:20.093 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:20.093 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:20.093 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
2023-03-03T16:56:22.040 INFO:tasks.rgw_multisite_tests:======================================================================
2023-03-03T16:56:22.040 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init
2023-03-03T16:56:22.041 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2023-03-03T16:56:22.042 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2023-03-03T16:56:22.042 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2023-03-03T16:56:22.043 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2023-03-03T16:56:22.043 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 1874, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init
2023-03-03T16:56:22.043 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2023-03-03T16:56:22.043 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2023-03-03T16:56:22.044 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2023-03-03T16:56:22.044 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_be0b7553d0f3ed10d2adf5e21bfc6f86773ca92a/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2023-03-03T16:56:22.044 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name
2023-03-03T16:56:22.044 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone1
2023-03-03T16:56:22.044 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------
</pre>