Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-03-13T10:35:57Z
Ceph
Redmine
rgw - Backport #64886 (In Progress): quincy: kafka: RGW hangs when broker is down for no persiste...
https://tracker.ceph.com/issues/64886
2024-03-13T10:35:57Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/56163">https://github.com/ceph/ceph/pull/56163</a></p>
rgw - Backport #64885 (In Progress): reef: kafka: RGW hangs when broker is down for no persistent...
https://tracker.ceph.com/issues/64885
2024-03-13T10:35:50Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/56158">https://github.com/ceph/ceph/pull/56158</a></p>
rgw - Bug #64710 (Pending Backport): kafka: RGW hangs when broker is down for no persistent notif...
https://tracker.ceph.com/issues/64710
2024-03-05T09:29:40Z
Yuval Lifshitz
yuvalif@yahoo.com
<p>based on this comment: <a class="external" href="https://github.com/ceph/ceph/pull/55051#issuecomment-1961950841">https://github.com/ceph/ceph/pull/55051#issuecomment-1961950841</a><br />the main issue is that the librdkafka timeout is set by defult to 5 minutes.<br />due to the 30 seconds idle timeout, we release the pending coroutines after 30 seconds.<br />but this is still causing the RGW to hang (as it exhaust all of its front-end connections).</p>
rgw - Bug #64184 (New): test_bn.py -v -a kafka_test: Fatal glibc error: tpp.c:87 (__pthread_tpp_c...
https://tracker.ceph.com/issues/64184
2024-01-25T14:41:04Z
Casey Bodley
cbodley@redhat.com
<pre>
2024-01-25T03:01:00.979 INFO:tasks.notification_tests:Running bucket-notifications-tests...
2024-01-25T03:01:00.979 DEBUG:teuthology.orchestra.run.smithi046:bucket notification tests against different endpoints> BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test
2024-01-25T03:01:11.896 INFO:tasks.rgw.client.0.smithi046.stdout:Fatal glibc error: tpp.c:87 (__pthread_tpp_change_priority): assertion failed: previous_prio == -1 || (previous_prio >= fifo_min_prio && previous_prio <= fifo_max_prio)
2024-01-25T03:01:11.896 INFO:tasks.rgw.client.0.smithi046.stdout:*** Caught signal (Aborted) **
2024-01-25T03:01:11.896 INFO:tasks.rgw.client.0.smithi046.stdout: in thread 7f69ab974640 thread_name:kafka_manager
2024-01-25T03:01:11.896 INFO:tasks.rgw.client.0.smithi046.stdout: ceph version 19.0.0-814-g1a8bb77b (1a8bb77be00267ce596e60f3b1141a4463aab767) squid (dev)
2024-01-25T03:01:11.896 INFO:tasks.rgw.client.0.smithi046.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f6add654db0]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 2: /lib64/libc.so.6(+0xa154c) [0x7f6add6a154c]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 3: raise()
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 4: abort()
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 5: /lib64/libc.so.6(+0x29130) [0x7f6add629130]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 6: /lib64/libc.so.6(+0x4daf7) [0x7f6add64daf7]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 7: /lib64/libc.so.6(+0xa7d18) [0x7f6add6a7d18]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 8: (std::_Function_handler<void (int), RGWPubSubKafkaEndpoint::send_to_completion_async(ceph::common::CephContext*, rgw_pubsub_s3_event const&, optional_yield)::{lambda(int)#1}>::_M_invoke(std::_Any_data const&, int&&)+0x95) [0x558c16fa84e5]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 9: (rgw::kafka::message_callback(rd_kafka_s*, rd_kafka_message_s const*, void*)+0xef) [0x558c170cc1ff]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 10: /lib64/librdkafka.so.1(+0x256ef) [0x7f6ade4776ef]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 11: /lib64/librdkafka.so.1(+0x5b862) [0x7f6ade4ad862]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 12: rd_kafka_poll()
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 13: (rgw::kafka::Manager::run()+0x350) [0x558c170ce5a0]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 14: /lib64/libstdc++.so.6(+0xdb924) [0x7f6addadb924]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 15: /lib64/libc.so.6(+0x9f802) [0x7f6add69f802]
2024-01-25T03:01:11.897 INFO:tasks.rgw.client.0.smithi046.stdout: 16: /lib64/libc.so.6(+0x3f450) [0x7f6add63f450]
</pre>
rgw - Bug #63786 (New): rados_cls_all: TestCls2PCQueue.MultiProducer hangs
https://tracker.ceph.com/issues/63786
2023-12-11T17:52:01Z
Laura Flores
<p>/a/yuriw-2023-12-07_16:39:06-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7482242<br /><pre><code class="text syntaxhl"><span class="CodeRay">2023-12-07T17:50:05.182 INFO:tasks.workunit.client.0.smithi086.stdout:[ RUN ] TestCls2PCQueue.MultiProducer
2023-12-07T17:50:09.511 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:00.507 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.513 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:00.509 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.533 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:00.529 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.557 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.554 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.562 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.559 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.565 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.516 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.566 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.517 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.583 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.580 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:09.586 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.537 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:29.555 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.551 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:29.558 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.554 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T17:50:29.576 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.572 86714== Warning: unimplemented fcntl command: 1036
...
2023-12-07T20:46:47.209 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:03:02:38.210 86714== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.240 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:03:02:38.197 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.242 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:03:02:38.199 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.245 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:03:02:38.247 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.248 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:03:02:38.250 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.252 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:03:02:38.209 86382== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:47.258 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:03:02:38.260 86718== Warning: unimplemented fcntl command: 1036
2023-12-07T20:46:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 124
2023-12-07T20:46:48.281 INFO:tasks.workunit:Stopping ['cls'] on client.0...
2023-12-07T20:46:48.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2023-12-07T20:46:48.585 ERROR:teuthology.run_tasks:Saw exception from tasks.
</span></code></pre></p>
rgw - Backport #63702 (In Progress): quincy: kafka crashed during message callback in teuthology
https://tracker.ceph.com/issues/63702
2023-11-30T14:07:09Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/54737">https://github.com/ceph/ceph/pull/54737</a></p>
rgw - Backport #63701 (In Progress): reef: kafka crashed during message callback in teuthology
https://tracker.ceph.com/issues/63701
2023-11-30T14:07:02Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/54736">https://github.com/ceph/ceph/pull/54736</a></p>
rgw - Backport #63631 (New): quincy: "test pushing kafka s3 notification on master" - no events a...
https://tracker.ceph.com/issues/63631
2023-11-24T09:37:43Z
Backport Bot
rgw - Backport #63630 (New): reef: "test pushing kafka s3 notification on master" - no events are...
https://tracker.ceph.com/issues/63630
2023-11-24T09:37:36Z
Backport Bot
rgw - Bug #63314 (Pending Backport): kafka crashed during message callback in teuthology
https://tracker.ceph.com/issues/63314
2023-10-25T08:16:11Z
Yuval Lifshitz
yuvalif@yahoo.com
<p>test: <a class="external" href="http://pulpito.front.sepia.ceph.com/yuvalif-2023-10-25_06:07:27-rgw:notifications-wip-topic-policy-distro-default-smithi/">http://pulpito.front.sepia.ceph.com/yuvalif-2023-10-25_06:07:27-rgw:notifications-wip-topic-policy-distro-default-smithi/</a><br />backtrace:<br /><pre>
Fatal glibc error: tpp.c:87 (__pthread_tpp_change_priority): assertion failed: previous_prio == -1 || (previous_prio >= fifo_min_prio && previous_prio <= fifo_max_prio)
*** Caught signal (Aborted) **
in thread 7fdf852cc640 thread_name:kafka_manager
ceph version 18.0.0-6835-ge5c0f496 (e5c0f4966b3da3fed8fcdae150335c900609e4d9) reef (dev)
1: /lib64/libc.so.6(+0x54db0) [0x7fe0b8054db0]
2: /lib64/libc.so.6(+0xa154c) [0x7fe0b80a154c]
3: raise()
4: abort()
5: /lib64/libc.so.6(+0x29130) [0x7fe0b8029130]
6: /lib64/libc.so.6(+0x4daf7) [0x7fe0b804daf7]
7: /lib64/libc.so.6(+0xa7d18) [0x7fe0b80a7d18]
8: (RGWPubSubKafkaEndpoint::Waiter::finish(int)+0xbe) [0x55dc941187ee]
9: (rgw::kafka::message_callback(rd_kafka_s*, rd_kafka_message_s const*, void*)+0xef) [0x55dc94245c8f]
10: /lib64/librdkafka.so.1(+0x256ef) [0x7fe0b90776ef]
11: /lib64/librdkafka.so.1(+0x5b862) [0x7fe0b90ad862]
12: rd_kafka_poll()
13: (rgw::kafka::Manager::run()+0x328) [0x55dc94248008]
14: /lib64/libstdc++.so.6(+0xdb924) [0x7fe0b84db924]
15: /lib64/libc.so.6(+0x9f802) [0x7fe0b809f802]
16: /lib64/libc.so.6(+0x3f450) [0x7fe0b803f450]
</pre></p>
rgw - Bug #62264 (Pending Backport): [ FAILED ] TestAMQP.IdleConnection (30132 ms)
https://tracker.ceph.com/issues/62264
2023-08-01T13:08:40Z
Casey Bodley
cbodley@redhat.com
<p>saw a failure in 'make check' on unrelated pr <a class="external" href="https://github.com/ceph/ceph/pull/47148">https://github.com/ceph/ceph/pull/47148</a>:<br /><pre>
215/285 Test #224: unittest_rgw_amqp .........................***Failed 35.29 sec
Running main() from gmock_main.cc
[==========] Running 23 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 23 tests from TestAMQP
[ RUN ] TestAMQP.ConnectionOK
[ OK ] TestAMQP.ConnectionOK (28 ms)
[ RUN ] TestAMQP.SSLConnectionOK
[ OK ] TestAMQP.SSLConnectionOK (12 ms)
[ RUN ] TestAMQP.PlainAndSSLConnectionsOK
[ OK ] TestAMQP.PlainAndSSLConnectionsOK (12 ms)
[ RUN ] TestAMQP.ConnectionReuse
[ OK ] TestAMQP.ConnectionReuse (10 ms)
[ RUN ] TestAMQP.NameResolutionFail
[ OK ] TestAMQP.NameResolutionFail (111 ms)
[ RUN ] TestAMQP.InvalidPort
[ OK ] TestAMQP.InvalidPort (112 ms)
[ RUN ] TestAMQP.InvalidHost
[ OK ] TestAMQP.InvalidHost (108 ms)
[ RUN ] TestAMQP.InvalidVhost
[ OK ] TestAMQP.InvalidVhost (108 ms)
[ RUN ] TestAMQP.UserPassword
[ OK ] TestAMQP.UserPassword (209 ms)
[ RUN ] TestAMQP.URLParseError
[ OK ] TestAMQP.URLParseError (114 ms)
[ RUN ] TestAMQP.ExchangeMismatch
[ OK ] TestAMQP.ExchangeMismatch (112 ms)
[ RUN ] TestAMQP.MaxConnections
[ OK ] TestAMQP.MaxConnections (392 ms)
[ RUN ] TestAMQP.ReceiveAck
[ OK ] TestAMQP.ReceiveAck (114 ms)
[ RUN ] TestAMQP.ImplicitConnectionClose
[ OK ] TestAMQP.ImplicitConnectionClose (629 ms)
[ RUN ] TestAMQP.ReceiveMultipleAck
[ OK ] TestAMQP.ReceiveMultipleAck (141 ms)
[ RUN ] TestAMQP.ReceiveAckForMultiple
[ OK ] TestAMQP.ReceiveAckForMultiple (209 ms)
[ RUN ] TestAMQP.DynamicCallback
[ OK ] TestAMQP.DynamicCallback (119 ms)
[ RUN ] TestAMQP.ReceiveNack
[ OK ] TestAMQP.ReceiveNack (120 ms)
[ RUN ] TestAMQP.FailWrite
[ OK ] TestAMQP.FailWrite (110 ms)
[ RUN ] TestAMQP.RetryInvalidHost
[ OK ] TestAMQP.RetryInvalidHost (704 ms)
[ RUN ] TestAMQP.RetryInvalidPort
[ OK ] TestAMQP.RetryInvalidPort (703 ms)
[ RUN ] TestAMQP.RetryFailWrite
[ OK ] TestAMQP.RetryFailWrite (703 ms)
[ RUN ] TestAMQP.IdleConnection
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:523: Failure
Expected equality of these values:
amqp::get_connection_count()
Which is: 1
connection_number
Which is: 0
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:75: Failure
Expected: (rc) < (0), actual: 0 vs 0
[ FAILED ] TestAMQP.IdleConnection (30132 ms)
[----------] 23 tests from TestAMQP (35012 ms total)
</pre></p>
rgw - Bug #62136 (Pending Backport): "test pushing kafka s3 notification on master" - no events a...
https://tracker.ceph.com/issues/62136
2023-07-24T14:23:49Z
Yuval Lifshitz
yuvalif@yahoo.com
<p>teuthology log: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/teuthology.log</a></p>
<pre>
FAIL: test pushing kafka s3 notification on master
...
no creation event found for key: <Key: ggozyw-1,0>
</pre>
<p>rgw log: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/remote/smithi129/log/rgw.ceph.client.0.log.gz">http://qa-proxy.ceph.com/teuthology/yuvalif-2023-07-23_21:15:40-rgw-wip-yuval-lua-test-distro-default-smithi/7349166/remote/smithi129/log/rgw.ceph.client.0.log.gz</a><br />does not show any kafka/notifications errors - could be a test timing issue.</p>
rgw - Backport #59498 (In Progress): reef: lua: crash when using Request.CopyFrom
https://tracker.ceph.com/issues/59498
2023-04-20T14:11:35Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/54296">https://github.com/ceph/ceph/pull/54296</a></p>
rgw - Backport #57561 (In Progress): quincy: pubsub test failures
https://tracker.ceph.com/issues/57561
2022-09-15T14:14:16Z
Backport Bot
<p><a class="external" href="https://github.com/ceph/ceph/pull/48914">https://github.com/ceph/ceph/pull/48914</a></p>
rgw - Bug #56572 (Pending Backport): pubsub test failures
https://tracker.ceph.com/issues/56572
2022-07-15T15:47:27Z
Casey Bodley
cbodley@redhat.com
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-12_13:41:13-rgw-wip-yuri-testing-2022-07-06-1021-quincy-distro-default-smithi/6926803/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-12_13:41:13-rgw-wip-yuri-testing-2022-07-06-1021-quincy-distro-default-smithi/6926803/teuthology.log</a></p>
<pre>
2022-07-14T09:30:45.606 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... ok
2022-07-14T09:30:45.607 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok
2022-07-14T09:30:45.608 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok
2022-07-14T09:30:45.610 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok
2022-07-14T09:30:45.611 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.612 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.613 INFO:tasks.rgw_multisite_tests:FAIL: test s3 records fetching
2022-07-14T09:30:45.614 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.614 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.615 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.615 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 773, in test_ps_s3_notification_records
2022-07-14T09:30:45.616 INFO:tasks.rgw_multisite_tests: verify_s3_records_by_elements(records, keys, exact_match=False)
2022-07-14T09:30:45.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements
2022-07-14T09:30:45.618 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.618 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-72,0>
2022-07-14T09:30:45.618 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.619 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.619 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of subscription
2022-07-14T09:30:45.620 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.620 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.621 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.622 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1291, in test_ps_subscription
2022-07-14T09:30:45.622 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False)
2022-07-14T09:30:45.622 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-07-14T09:30:45.623 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.623 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-78,0>
2022-07-14T09:30:45.624 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.624 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.624 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands
2022-07-14T09:30:45.625 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.625 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.626 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1402, in test_ps_admin
2022-07-14T09:30:45.627 INFO:tasks.rgw_multisite_tests: verify_events_by_elements({"events": events1+events2}, keys, exact_match=False)
2022-07-14T09:30:45.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-07-14T09:30:45.627 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.628 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-79,0>
2022-07-14T09:30:45.628 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.629 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.629 INFO:tasks.rgw_multisite_tests:FAIL: test that events are only sent on incremental sync
2022-07-14T09:30:45.630 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.630 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.630 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.631 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1474, in test_ps_incremental_sync
2022-07-14T09:30:45.631 INFO:tasks.rgw_multisite_tests: assert_equal(count, number_of_objects)
2022-07-14T09:30:45.632 INFO:tasks.rgw_multisite_tests:AssertionError: 0 != 10
2022-07-14T09:30:45.632 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.632 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.633 INFO:tasks.rgw_multisite_tests:FAIL: test subscriptions for different events
2022-07-14T09:30:45.633 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.634 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.634 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.634 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1552, in test_ps_event_type_subscription
2022-07-14T09:30:45.635 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False)
2022-07-14T09:30:45.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-07-14T09:30:45.636 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.636 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-81,0>
2022-07-14T09:30:45.637 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.637 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.637 INFO:tasks.rgw_multisite_tests:FAIL: test incremental fetching of events from a subscription
2022-07-14T09:30:45.638 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.638 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.639 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1670, in test_ps_event_fetching
2022-07-14T09:30:45.640 INFO:tasks.rgw_multisite_tests: verify_events_by_elements({'events': all_events}, keys, exact_match=False)
2022-07-14T09:30:45.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-07-14T09:30:45.640 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.641 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-82,0>
2022-07-14T09:30:45.641 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.641 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.642 INFO:tasks.rgw_multisite_tests:FAIL: test acking of some events in a subscription
2022-07-14T09:30:45.642 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.643 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.643 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1720, in test_ps_event_acking
2022-07-14T09:30:45.644 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False)
2022-07-14T09:30:45.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-07-14T09:30:45.645 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.645 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-83,0>
2022-07-14T09:30:45.645 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.646 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.646 INFO:tasks.rgw_multisite_tests:FAIL: test object creation notifications in using put/copy/post
2022-07-14T09:30:45.647 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.647 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.647 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.648 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1797, in test_ps_creation_triggers
2022-07-14T09:30:45.648 INFO:tasks.rgw_multisite_tests: assert len(events['events']) >= 3
2022-07-14T09:30:45.649 INFO:tasks.rgw_multisite_tests:AssertionError
2022-07-14T09:30:45.649 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.650 INFO:tasks.rgw_multisite_tests:======================================================================
2022-07-14T09:30:45.650 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion
2022-07-14T09:30:45.650 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.651 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-07-14T09:30:45.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-07-14T09:30:45.651 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2022-07-14T09:30:45.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 2217, in test_ps_delete_bucket
2022-07-14T09:30:45.652 INFO:tasks.rgw_multisite_tests: verify_s3_records_by_elements(records, keys, exact_match=False)
2022-07-14T09:30:45.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_9259dd13cbf208472db2e03b3fd84db4f39c53bc/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements
2022-07-14T09:30:45.653 INFO:tasks.rgw_multisite_tests: assert False, err
2022-07-14T09:30:45.653 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: kmvefe-86,0>
2022-07-14T09:30:45.654 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.654 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-07-14T09:30:45.655 INFO:tasks.rgw_multisite_tests:Ran 56 tests in 1954.338s
2022-07-14T09:30:45.655 INFO:tasks.rgw_multisite_tests:
2022-07-14T09:30:45.655 INFO:tasks.rgw_multisite_tests:FAILED (failures=9)
</pre>