Ceph » rgw

Ceph version: 16.2.13
Steps to reproduce in boto3:
1. Create a topic and notification

#!/usr/bin/python3
import boto3
import urllib.parse
import logging
import sys
from boto3.s3.transfer import TransferConfig

#boto3.set_stream_logger('botocore',logging.DEBUG)
ep = 'https://huynnp'
#ep = 'http://10.5.89.163:8080'
access_key = 'D25YEUWC9WFV0XU6FAOP'
secret_key = '1Xm14nl8LzM3zHhZZxP3kg0NQxpAuXkIoImJTtVV'

s3 = boto3.resource('s3', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, use_ssl=True, verify=False)

s3client = boto3.client('s3', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, use_ssl=True, verify=False)

snsclient = boto3.client('sns', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, region_name = "hn", config=Config(signature_version='s3'), use_ssl=True, verify=False)

bucket = 'huybucket-notify-bucket-lab2'
s3.create_bucket(Bucket=bucket)

1. create topic
   topic_name = 'huynnp-topic'
   endpoint_args = 'push-endpoint=kafka://10.5.90.159:9092'
   attributes = {nvp⁰ : nvp¹ for nvp in urllib.parse.parse_qsl(endpoint_args, keep_blank_values=True)}
   response = snsclient.create_topic(Name=topic_name, Attributes=attributes)

#create bucket notification
bucket_notification = s3.BucketNotification(bucket)
response = bucket_notification.put(
NotificationConfiguration={
'TopicConfigurations': [ {
'Id': f'{bucket}',
'TopicArn': f'arn:aws:sns:hn::{topic_name}',
'Events': [
's3:ObjectCreated:*',
's3:ObjectRemoved:*'
],
'Filter': {
'Key': {
'FilterRules': [ {
'Name': 'prefix',
'Value': ''
},
]
}
}
},
],
},
)
print(response)

2. Try to put an object and delete it using delete_object, notifications will be published normally
s3.Object(bucket, 'testfile9').put(Body="0"*5)
s3client.delete_object(
Bucket=bucket,
Key='testfile9')

3. Try to put an object and delete it using delete_objects, Radosgw will crash when trying to publish event s3:ObjectRemoved:Delete
s3.Object(bucket, 'testfile9').put(Body="0"*5)
s3client.delete_objects(
Bucket=bucket,
Delete={
'Objects': [ {
'Key': 'testfile9',
},
],
})

Radosgw debug log:

2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete init op
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op mask
2023-06-12T10:42:27.494+0700 7f26c5004700 20 req 14384231648107788089 0.003999959s s3:multi_object_delete required_mask= 4 user.op_mask=7
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op permissions
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete NOTICE: call to do_aws4_auth_completion
2023-06-12T10:42:27.494+0700 7f26c5004700 10 req 14384231648107788089 0.003999959s s3:multi_object_delete v4 auth ok -- do_aws4_auth_completion
2023-06-12T10:42:27.494+0700 7f26c5004700 20 req 14384231648107788089 0.003999959s s3:multi_object_delete -- Getting permissions begin with perm_mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0) mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for uid=huy
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Found permission: 15
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for group=1 mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Permissions for group not found
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for group=2 mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Permissions for group not found
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete - Getting permissions done for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0), owner=huy, perm=2
2023-06-12T10:42:27.494+0700 7f26c5004700 10 req 14384231648107788089 0.003999959s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op params
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete pre-executing
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete executing
2023-06-12T10:42:27.495+0700 7f26c5004700 20 req 14384231648107788089 0.004999949s s3:multi_object_delete get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s>prefetch_data=0
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s s3:multi_object_delete INFO: notification: 'huybucket-notify-bucket-lab2' on topic: 'huynnp-topic' and bucket: 'huybucket-notify-bucket-lab2' (unique topic: 'huybucket-notify-bucket-lab2_huynnp-topic') apply to event of type: 's3:ObjectRemoved:Delete'
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s s3:multi_object_delete get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s->prefetch_data=0
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s->prefetch_data=0
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable), process radosgw, pid 3932288
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 framework: beast
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 framework conf key: port, val: 8080
2023-06-12T10:42:34.120+0700 7fc31418b5c0 1 radosgw_Main not setting numa affinity
2023-06-12T10:42:34.124+0700 7fc2fa8df700 20 reqs_thread_entry: start
2023-06-12T10:42:34.124+0700 7fc2fa0de700 10 entry start

Here is my Kafka server.properties if anyone want to check it:

listeners=PLAINTEXT://10.5.90.159:9092,SSL://10.5.90.159:9093
advertised.listeners=PLAINTEXT://10.5.90.159:9092,SSL://10.5.90.159:9093
security.inter.broker.protocol=PLAINTEXT
security.protocol=PLAINTEXT,SSL
ssl.keystore.location=/path/to/KeyStore.jks
ssl.keystore.password=xxxxxx
ssl.key.password=xxxxxx
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxx