Project

General

Profile

Actions
Copy link

Bug #65463

open

rgw/notifications: test data path v2 persistent migration fails

Added by Casey Bodley 18 days ago. Updated 15 days ago.

Status:
New
Priority:
Urgent
Assignee:
Yuval Lifshitz
Target version:
-
% Done:

0%

Source:
Tags:
notifications
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

from https://qa-proxy.ceph.com/teuthology/cbodley-2024-04-12_12:44:47-rgw-wip-rgw-account-v3-distro-default-smithi/7653870/teuthology.log (testing the wip-rgw-account-v3 branch)

2024-04-12T13:55:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:======================================================================
2024-04-12T13:55:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:FAIL: test data path v2 persistent migration
2024-04-12T13:55:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------------------------
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:Traceback (most recent call last):
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:    self.test(*self.arg)
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:  File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 4758, in test_persistent_ps_s3_data_path_v2_migration
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:    assert False, str(e)
2024-04-12T13:55:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:AssertionError: 20 != 0
2024-04-12T13:55:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:-------------------- >> begin captured logging << --------------------

this assertion fails in wait_for_queue_to_drain() after logging this WARNING:

2024-04-12T13:55:09.138 INFO:teuthology.orchestra.run.smithi134.stderr:bucket_notification.tests: DEBUG: running command: /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/../test-rgw-call.sh call_rgw_admin noname topic stats --topic lpxekd-22_topic
2024-04-12T13:55:09.138 INFO:teuthology.orchestra.run.smithi134.stderr:bucket_notification.tests: DEBUG: running command: /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/../test-rgw-call.sh call_rgw_admin noname topic stats --topic lpxekd-22_topic
2024-04-12T13:55:09.138 INFO:teuthology.orchestra.run.smithi134.stderr:bucket_notification.tests: DEBUG: running command: /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/../test-rgw-call.sh call_rgw_admin noname topic stats --topic lpxekd-22_topic
2024-04-12T13:55:09.139 INFO:teuthology.orchestra.run.smithi134.stderr:bucket_notification.test_bn: WARNING: queue lpxekd-22_topic still has 20 entries after 154s

looking through the rgw log, i see it trying/failing to send these notifications. from https://qa-proxy.ceph.com/teuthology/cbodley-2024-04-12_12:44:47-rgw-wip-rgw-account-v3-distro-default-smithi/7653870/remote/smithi134/log/rgw.ceph.client.0.log.gz:

2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: msg->data.result=56 req_data->id=198538 http_status=0
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: curl error: Failure when receiving data from the peer req_data->error_buf=Recv failure: Connection reset by peer
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: msg->data.result=56 req_data->id=198539 http_status=0
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: curl error: Failure when receiving data from the peer req_data->error_buf=Recv failure: Connection reset by peer
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: msg->data.result=56 req_data->id=198540 http_status=0
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: curl error: Failure when receiving data from the peer req_data->error_buf=Recv failure: Connection reset by peer
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: msg->data.result=56 req_data->id=198542 http_status=0
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: curl error: Failure when receiving data from the peer req_data->error_buf=Recv failure: Connection reset by peer
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: msg->data.result=56 req_data->id=198543 http_status=0
2024-04-12T13:51:57.127+0000 7fd569b8e640 20 ERROR: curl error: Failure when receiving data from the peer req_data->error_buf=Recv failure: Connection reset by peer

Actions
Copy link
 #1

Updated by Casey Bodley 18 days ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #2

Updated by Yuval Lifshitz 15 days ago

  • even tough no crash is observed, it seems like a similar issue to: https://tracker.ceph.com/issues/65337. when running locally, i see the same issue without a crash
  • the failed curl messages in the log could be intentional in the "test_ps_s3_persistent_multiple_endpoints" test
Actions
Copy link

Also available in: Atom PDF