Project

General

Profile

Actions

Bug #63335

open

Notification kafka: Persistent messages are removed even when the broker is down and no ack is received

Added by Krunal Chheda 6 months ago. Updated about 1 month ago.

Status:
Pending Backport
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
notification backport_processed
Backport:
squid reef quincy
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is the regression caused due to changes in https://github.com/ceph/ceph/pull/51864.

Currently the way Kafka processing works when the ACK level is set broker, is notification is sent to kafka using the rd_kafka_produce(), which currently DOES not return -1 if the BROKER is down.
But when the broker is down, the CALLBACK from kafka acknowledgement is never returned and hence this ack is propagated back to notification system which then does a retry and does not delete the persistent notification from queue.

But after the https://github.com/ceph/ceph/pull/51864, the connection was destroyed after idle_timeout and callbacks were invoked with SUCCESS, so even tho the notification were not delivered, they were removed from the queue because the callback was invoked with sucess.


Related issues 3 (2 open1 closed)

Copied to rgw - Backport #64792: reef: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedIn ProgressYuval LifshitzActions
Copied to rgw - Backport #64793: quincy: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedIn ProgressYuval LifshitzActions
Copied to rgw - Backport #64861: squid: Notification kafka: Persistent messages are removed even when the broker is down and no ack is receivedRejectedYuval LifshitzActions
Actions #1

Updated by Casey Bodley 5 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Yuval Lifshitz
Actions #2

Updated by Casey Bodley about 1 month ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to reef quincy
Actions #3

Updated by Backport Bot about 1 month ago

  • Copied to Backport #64792: reef: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received added
Actions #4

Updated by Backport Bot about 1 month ago

  • Copied to Backport #64793: quincy: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received added
Actions #5

Updated by Backport Bot about 1 month ago

  • Tags changed from notification to notification backport_processed
Actions #6

Updated by Yuval Lifshitz about 1 month ago

  • Tags changed from notification backport_processed to notification
  • Backport changed from reef quincy to squid reef quincy
Actions #7

Updated by Backport Bot about 1 month ago

  • Copied to Backport #64861: squid: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received added
Actions #8

Updated by Backport Bot about 1 month ago

  • Tags changed from notification to notification backport_processed
Actions

Also available in: Atom PDF