Bug #65590
openrgw_multi.tests.test_topic_notification_sync: PutBucketNotificationConfiguration fails with ConcurrentModification
0%
Description
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_topic_notification_sync 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5d4ca1d4687229e2ba4144f00b14e7c911ce18dd/qa/../src/test/rgw/rgw_multi/tests.py", line 1788, in test_topic_notification_sync 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: zone_conn.create_notification(bucket.name, [topic_conf]) 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5d4ca1d4687229e2ba4144f00b14e7c911ce18dd/qa/tasks/rgw_multi/zone_rados.py", line 160, in create_notification 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: return self.s3_client.put_bucket_notification_configuration( 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: return self._make_api_call(operation_name, kwargs) 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests: raise error_class(parsed_response, operation_name) 2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:botocore.exceptions.ClientError: An error occurred (ConcurrentModification) when calling the PutBucketNotificationConfiguration operation: None
this is an ECANCELED error coming from our write of the bucket instance metadata. it's a common error for metadata writes on secondary zones, because our write races with metadata sync from the write that we forwarded to the master zone
rgw ops should generally retry these writes on ECANCELED after reading the updated bucket metadata. see a very simple example from RGWPutCORS::execute(): https://github.com/ceph/ceph/blob/c46a99d/src/rgw/rgw_op.cc#L6058-L6061
for RGWPSCreateNotifOp::execute_v2(), the entire block of https://github.com/ceph/ceph/blob/c46a99d/src/rgw/rgw_rest_pubsub.cc#L1327-L1362 should be done inside of retry_raced_bucket_write()
i guess that means the retry_raced_bucket_write() template needs to move from rgw_op.cc to rgw_op.h or some other header
Updated by Casey Bodley 9 days ago
- Status changed from New to Fix Under Review
- Assignee set to Krunal Chheda
- Pull request ID set to 57004
Updated by Casey Bodley 2 days ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 2 days ago
- Copied to Backport #65767: squid: rgw_multi.tests.test_topic_notification_sync: PutBucketNotificationConfiguration fails with ConcurrentModification added
Updated by Backport Bot 2 days ago
- Tags changed from multisite notifications to multisite notifications backport_processed