Bug #61640
closedBug #63580: notifications: sending notifications with multidelete is causing a crash
RGW Kafka: Radosgw crashes when DeleteMultiObj is executed on a bucket that has enabled notifications.
0%
Description
Ceph version: 16.2.13
Steps to reproduce in boto3:
1. Create a topic and notification
#!/usr/bin/python3
import boto3
import urllib.parse
import logging
import sys
from boto3.s3.transfer import TransferConfig
#boto3.set_stream_logger('botocore',logging.DEBUG)
ep = 'https://huynnp'
#ep = 'http://10.5.89.163:8080'
access_key = 'D25YEUWC9WFV0XU6FAOP'
secret_key = '1Xm14nl8LzM3zHhZZxP3kg0NQxpAuXkIoImJTtVV'
s3 = boto3.resource('s3', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, use_ssl=True, verify=False)
s3client = boto3.client('s3', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, use_ssl=True, verify=False)
snsclient = boto3.client('sns', endpoint_url=ep,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key, region_name = "hn", config=Config(signature_version='s3'), use_ssl=True, verify=False)
bucket = 'huybucket-notify-bucket-lab2'
s3.create_bucket(Bucket=bucket)
- create topic
topic_name = 'huynnp-topic'
endpoint_args = 'push-endpoint=kafka://10.5.90.159:9092'
attributes = {nvp0 : nvp1 for nvp in urllib.parse.parse_qsl(endpoint_args, keep_blank_values=True)}
response = snsclient.create_topic(Name=topic_name, Attributes=attributes)
#create bucket notification
bucket_notification = s3.BucketNotification(bucket)
response = bucket_notification.put(
NotificationConfiguration={
'TopicConfigurations': [
{
'Id': f'{bucket}',
'TopicArn': f'arn:aws:sns:hn::{topic_name}',
'Events': [
's3:ObjectCreated:*',
's3:ObjectRemoved:*'
],
'Filter': {
'Key': {
'FilterRules': [
{
'Name': 'prefix',
'Value': ''
},
]
}
}
},
],
},
)
print(response)
2. Try to put an object and delete it using delete_object, notifications will be published normally
s3.Object(bucket, 'testfile9').put(Body="0"*5)
s3client.delete_object(
Bucket=bucket,
Key='testfile9')
3. Try to put an object and delete it using delete_objects, Radosgw will crash when trying to publish event s3:ObjectRemoved:Delete
s3.Object(bucket, 'testfile9').put(Body="0"*5)
s3client.delete_objects(
Bucket=bucket,
Delete={
'Objects': [
{
'Key': 'testfile9',
},
],
})
Radosgw debug log:
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete init op
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op mask
2023-06-12T10:42:27.494+0700 7f26c5004700 20 req 14384231648107788089 0.003999959s s3:multi_object_delete required_mask= 4 user.op_mask=7
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op permissions
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete NOTICE: call to do_aws4_auth_completion
2023-06-12T10:42:27.494+0700 7f26c5004700 10 req 14384231648107788089 0.003999959s s3:multi_object_delete v4 auth ok -- do_aws4_auth_completion
2023-06-12T10:42:27.494+0700 7f26c5004700 20 req 14384231648107788089 0.003999959s s3:multi_object_delete -- Getting permissions begin with perm_mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0) mask=50 Getting permissions done for identity=rgw::auth::SysReqApplier
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for uid=huy
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Found permission: 15
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for group=1 mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Permissions for group not found
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Searching permissions for group=2 mask=50
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete Permissions for group not found
2023-06-12T10:42:27.494+0700 7f26c5004700 5 req 14384231648107788089 0.003999959s s3:multi_object_delete -> rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0), owner=huy, perm=2>prefetch_data=0
2023-06-12T10:42:27.494+0700 7f26c5004700 10 req 14384231648107788089 0.003999959s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=huy, acct_name=huy user, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete verifying op params
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete pre-executing
2023-06-12T10:42:27.494+0700 7f26c5004700 2 req 14384231648107788089 0.003999959s s3:multi_object_delete executing
2023-06-12T10:42:27.495+0700 7f26c5004700 20 req 14384231648107788089 0.004999949s s3:multi_object_delete get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s s3:multi_object_delete INFO: notification: 'huybucket-notify-bucket-lab2' on topic: 'huynnp-topic' and bucket: 'huybucket-notify-bucket-lab2' (unique topic: 'huybucket-notify-bucket-lab2_huynnp-topic') apply to event of type: 's3:ObjectRemoved:Delete'
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s s3:multi_object_delete get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s->prefetch_data=0
2023-06-12T10:42:27.500+0700 7f26b87eb700 20 req 14384231648107788089 0.009999898s get_obj_state: rctx=0x7f271b85da30 obj=huybucket-notify-bucket-lab2:testfile9 state=0x559126d533e8 s->prefetch_data=0
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable), process radosgw, pid 3932288
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 framework: beast
2023-06-12T10:42:34.120+0700 7fc31418b5c0 0 framework conf key: port, val: 8080
2023-06-12T10:42:34.120+0700 7fc31418b5c0 1 radosgw_Main not setting numa affinity
2023-06-12T10:42:34.124+0700 7fc2fa8df700 20 reqs_thread_entry: start
2023-06-12T10:42:34.124+0700 7fc2fa0de700 10 entry start
Here is my Kafka server.properties if anyone want to check it:
listeners=PLAINTEXT://10.5.90.159:9092,SSL://10.5.90.159:9093
advertised.listeners=PLAINTEXT://10.5.90.159:9092,SSL://10.5.90.159:9093
security.inter.broker.protocol=PLAINTEXT
security.protocol=PLAINTEXT,SSL
ssl.keystore.location=/path/to/KeyStore.jks
ssl.keystore.password=xxxxxx
ssl.key.password=xxxxxx
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxx
Updated by Casey Bodley 11 months ago
- Assignee set to Yuval Lifshitz
- Tags set to multidelete notifications kafka
Updated by Yuval Lifshitz 6 months ago
- Assignee changed from Yuval Lifshitz to Huy Nguyen
could you please see if this is happening in "quincy" or "reef"?
also, in the rge log you should see the crash backtrace. could you please pate that here?
Updated by Yuval Lifshitz 6 months ago
- Status changed from New to Duplicate
- Parent task set to #63580