Project

General

Profile

Actions
Copy link

Bug #63335

open

Notification kafka: Persistent messages are removed even when the broker is down and no ack is received

Added by Krunal Chheda 7 months ago. Updated 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Yuval Lifshitz
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
notification backport_processed
Backport:
squid reef quincy
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
54215
Crash signature (v1):
Crash signature (v2):

Description

This is the regression caused due to changes in https://github.com/ceph/ceph/pull/51864.

Currently the way Kafka processing works when the ACK level is set broker, is notification is sent to kafka using the rd_kafka_produce(), which currently DOES not return -1 if the BROKER is down.
But when the broker is down, the CALLBACK from kafka acknowledgement is never returned and hence this ack is propagated back to notification system which then does a retry and does not delete the persistent notification from queue.

But after the https://github.com/ceph/ceph/pull/51864, the connection was destroyed after idle_timeout and callbacks were invoked with SUCCESS, so even tho the notification were not delivered, they were removed from the queue because the callback was invoked with sucess.

Related issues 3 (1 open — 2 closed)

Copied to rgw - Backport #64792: reef: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedResolvedYuval LifshitzActions
Copied to rgw - Backport #64793: quincy: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedIn ProgressYuval LifshitzActions
Copied to rgw - Backport #64861: squid: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedRejectedYuval LifshitzActions
Actions
Copy link

Also available in: Atom PDF