Ceph » rgw

2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:======================================================================
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_topic_notification_sync
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2024-04-18T20:15:45.433 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5d4ca1d4687229e2ba4144f00b14e7c911ce18dd/qa/../src/test/rgw/rgw_multi/tests.py", line 1788, in test_topic_notification_sync
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:    zone_conn.create_notification(bucket.name, [topic_conf])
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5d4ca1d4687229e2ba4144f00b14e7c911ce18dd/qa/tasks/rgw_multi/zone_rados.py", line 160, in create_notification
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:    return self.s3_client.put_bucket_notification_configuration(
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:    return self._make_api_call(operation_name, kwargs)
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:    raise error_class(parsed_response, operation_name)
2024-04-18T20:15:45.434 INFO:tasks.rgw_multisite_tests:botocore.exceptions.ClientError: An error occurred (ConcurrentModification) when calling the PutBucketNotificationConfiguration operation: None

this is an ECANCELED error coming from our write of the bucket instance metadata. it's a common error for metadata writes on secondary zones, because our write races with metadata sync from the write that we forwarded to the master zone

rgw ops should generally retry these writes on ECANCELED after reading the updated bucket metadata. see a very simple example from RGWPutCORS::execute(): https://github.com/ceph/ceph/blob/c46a99d/src/rgw/rgw_op.cc#L6058-L6061

for RGWPSCreateNotifOp::execute_v2(), the entire block of https://github.com/ceph/ceph/blob/c46a99d/src/rgw/rgw_rest_pubsub.cc#L1327-L1362 should be done inside of retry_raced_bucket_write()

i guess that means the retry_raced_bucket_write() template needs to move from rgw_op.cc to rgw_op.h or some other header