Support #44436
Ceph BucketNotifications do not push messages to Kafka
% Done:
0%
Tags:
Reviewed:
Affected Versions:
Pull request ID:
Description
I have ceph three-nodes cluster for tests with radosgw on first node.
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# cat /etc/ceph/ceph.conf [global] fsid = a204e3b5-b0e1-4598-aadc-f46011b562f0 mon_initial_members = ceph-test-01, ceph-test-02, ceph-test-03 mon_host = 172.19.19.252,172.19.19.251,172.19.19.244 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public network = 172.19.19.0/24 rgw_frontends = beast port=80
And Kafka broker for receiving messages.
I've create topic by docs (https://docs.ceph.com/docs/master/radosgw/notifications/).
POST http://ceph-test-01/?Action=CreateTopic&Name=bucketnotifications&push-endpoint=kafka://kafka-test-01:9092 with S3 credentials Response: <CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <CreateTopicResult> <TopicArn>arn:aws:sns:us::bucketnotifications</TopicArn> </CreateTopicResult> <ResponseMetadata> <RequestId>14804357-9c29-437d-9789-c127e1c13d23.44108.275</RequestId> </ResponseMetadata> </CreateTopicResponse>
I've created bucket "new-bucket" and checked its configuration
GET http://ceph-test-01/new-bucket?notification Response: <NotificationConfiguration> <TopicConfiguration> <Id>bucketnotifications</Id> <Topic>arn:aws:sns:us::bucketnotifications</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration>
When I add files to bucket or remove them, ceph do not push notifications to Kafka. Radosgw logs(in debug mode) contains next strings:
2020-03-04 10:58:00.533 7f29a8da3700 20 notification: 'bucketnotifications' on topic: 'bucketnotifications' and bucket: 'new-bucket' (unique topic: 'bucketnotifications') apply to event of type: 's3:ObjectCreated:Put' 2020-03-04 10:58:00.533 7f29a8da3700 1 ERROR: failed to create push endpoint: kafka://kafka-test-01:9092 due to: pubsub endpoint configuration error: unknown schema in: kafka://kafka-test-01:9092 2020-03-04 10:58:00.533 7f29a8da3700 5 req 188 0.248s s3:put_obj WARNING: publishing notification failed, with error: -22
Kafka has no new topics.
History
#1 Updated by Greg Farnum over 3 years ago
- Project changed from Ceph to rgw
- Category deleted (
librados)
#2 Updated by Yuval Lifshitz 8 months ago
- Status changed from New to Closed
nautilus not supprted anymore. cannot reproduce in newer versions