Bug #63335: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received - rgw - Ceph

Actions

Copy link

Bug #63335

open

Notification kafka: Persistent messages are removed even when the broker is down and no ack is received

Added by Krunal Chheda 6 months ago. Updated about 2 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Yuval Lifshitz

Target version:

% Done:

Source:

Community (dev)

Tags:

notification backport_processed

Backport:

squid reef quincy

Regression:

Yes

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

54215

Crash signature (v1):

Crash signature (v2):

Description

This is the regression caused due to changes in https://github.com/ceph/ceph/pull/51864.

Currently the way Kafka processing works when the ACK level is set broker, is notification is sent to kafka using the rd_kafka_produce(), which currently DOES not return -1 if the BROKER is down.
But when the broker is down, the CALLBACK from kafka acknowledgement is never returned and hence this ack is propagated back to notification system which then does a retry and does not delete the persistent notification from queue.

But after the https://github.com/ceph/ceph/pull/51864, the connection was destroyed after idle_timeout and callbacks were invoked with SUCCESS, so even tho the notification were not delivered, they were removed from the queue because the callback was invoked with sucess.

Related issues 3 (2 open — 1 closed)

Actions

Copy link

Updated by Casey Bodley 6 months ago

Status changed from New to Fix Under Review
Assignee set to Yuval Lifshitz

Actions

Copy link

Updated by Casey Bodley about 2 months ago

Status changed from Fix Under Review to Pending Backport
Backport set to reef quincy

Actions

Copy link

Updated by Backport Bot about 2 months ago

Copied to Backport #64792: reef: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received added

Actions

Copy link

Updated by Backport Bot about 2 months ago

Copied to Backport #64793: quincy: Notification kafka: Persistent messages are removed even when the broker is down and no ack is received added

Actions

Copy link