Project

General

Profile

Actions
Copy link

Bug #62136

open

"test pushing kafka s3 notification on master" - no events are sent

Added by Yuval Lifshitz 9 months ago. Updated 2 days ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Yuval Lifshitz
Target version:
-
% Done:

0%

Source:
Development
Tags:
notifications backport_processed
Backport:
reef, quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
54637
Crash signature (v1):
Crash signature (v2):

Description

teuthology log: http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/teuthology.log

FAIL: test pushing kafka s3 notification on master
...
no creation event found for key: <Key: ggozyw-1,0>

rgw log: http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/remote/smithi129/log/rgw.ceph.client.0.log.gz
does not show any kafka/notifications errors - could be a test timing issue.

Related issues 2 (2 open — 0 closed)

Copied to rgw - Backport #63630: reef: "test pushing kafka s3 notification on master" - no events are sentNewYuval LifshitzActions
Copied to rgw - Backport #63631: quincy: "test pushing kafka s3 notification on master" - no events are sentNewYuval LifshitzActions
Actions
Copy link
 #3

Updated by Casey Bodley 5 months ago

  • Subject changed from kafka test failure - no events are sent to "test pushing kafka s3 notification on master" - no events are sent
Actions
Copy link
 #4

Updated by Yuval Lifshitz 5 months ago

  • Status changed from New to Fix Under Review
  • Backport changed from reef to reef, quincy
  • Pull request ID set to 54637
Actions
Copy link
 #5

Updated by Yuval Lifshitz 5 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #6

Updated by Backport Bot 5 months ago

  • Copied to Backport #63630: reef: "test pushing kafka s3 notification on master" - no events are sent added
Actions
Copy link
 #7

Updated by Backport Bot 5 months ago

  • Copied to Backport #63631: quincy: "test pushing kafka s3 notification on master" - no events are sent added
Actions
Copy link
 #8

Updated by Backport Bot 5 months ago

  • Tags changed from notifications to notifications backport_processed
Actions
Copy link
 #9

Updated by Casey Bodley 3 days ago

this says resolved, but i still see failures like this on main:

2024-04-24T14:12:21.427 INFO:teuthology.orchestra.run.smithi037.stderr:======================================================================
2024-04-24T14:12:21.427 INFO:teuthology.orchestra.run.smithi037.stderr:FAIL: test pushing kafka s3 notification on master
2024-04-24T14:12:21.427 INFO:teuthology.orchestra.run.smithi037.stderr:----------------------------------------------------------------------
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:Traceback (most recent call last):
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:  File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/case.py", line 198, in runTest
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:    self.test(*self.arg)
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:  File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 1488, in test_ps_s3_notification_push_kafka_on_master
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:    assert False, str(e)
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:AssertionError: no creation event found for key: <Key: cyorcm-1,3>
2024-04-24T14:12:21.429 INFO:teuthology.orchestra.run.smithi037.stderr:-------------------- >> begin captured logging << --------------------

ex https://qa-proxy.ceph.com/teuthology/cbodley-2024-04-24_12:59:55-rgw-wip-cbodley-testing-distro-default-smithi/7671454/teuthology.log

Actions
Copy link
 #10

Updated by Yuval Lifshitz 2 days ago

could be another issue with the kafka consumer (on top of what was fixed in PR 54637):

2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Kafka consumer created on topic: cyorcm-1_topic_1
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Kafka receiver started
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.484 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.485 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out
2024-04-24T14:12:21.485 INFO:teuthology.orchestra.run.smithi037.stdout:Error: timed out

Actions
Copy link
 #11

Updated by Yuval Lifshitz 2 days ago

created a separate tracker: https://tracker.ceph.com/issues/65662

Actions
Copy link

Also available in: Atom PDF