Project

General

Profile

Bug #42042

unittest_rgw_amqp failing in master

Added by Kefu Chai 3 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

172/186 Test #176: unittest_rgw_amqp .......................***Failed    6.27 sec
Running main() from gmock_main.cc
[==========] Running 21 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 21 tests from TestAMQP
[ RUN      ] TestAMQP.ConnectionOK
[       OK ] TestAMQP.ConnectionOK (101 ms)
[ RUN      ] TestAMQP.ConnectionReuse
[       OK ] TestAMQP.ConnectionReuse (2 ms)
[ RUN      ] TestAMQP.NameResolutionFail
[       OK ] TestAMQP.NameResolutionFail (104 ms)
[ RUN      ] TestAMQP.InvalidPort
[       OK ] TestAMQP.InvalidPort (2 ms)
[ RUN      ] TestAMQP.InvalidHost
[       OK ] TestAMQP.InvalidHost (3 ms)
[ RUN      ] TestAMQP.InvalidVhost
[       OK ] TestAMQP.InvalidVhost (2 ms)
[ RUN      ] TestAMQP.UserPassword
[       OK ] TestAMQP.UserPassword (105 ms)
[ RUN      ] TestAMQP.URLParseError
[       OK ] TestAMQP.URLParseError (101 ms)
[ RUN      ] TestAMQP.ExchangeMismatch
[       OK ] TestAMQP.ExchangeMismatch (109 ms)
[ RUN      ] TestAMQP.MaxConnections
[       OK ] TestAMQP.MaxConnections (505 ms)
[ RUN      ] TestAMQP.ReceiveAck
[       OK ] TestAMQP.ReceiveAck (321 ms)
[ RUN      ] TestAMQP.ReceiveMultipleAck
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:250: Failure
Expected equality of these values:
  callbacks_invoked
    Which is: 76
  NUMBER_OF_CALLS
    Which is: 100
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
[  FAILED  ] TestAMQP.ReceiveMultipleAck (315 ms)
[ RUN      ] TestAMQP.ReceiveAckForMultiple
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:269: Failure
Expected equality of these values:
  callbacks_invoked
    Which is: 84
  NUMBER_OF_CALLS
    Which is: 100
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:204: Failure
Expected equality of these values:
  0
  rc
    Which is: -4098
[  FAILED  ] TestAMQP.ReceiveAckForMultiple (302 ms)
[ RUN      ] TestAMQP.DynamicCallback
[       OK ] TestAMQP.DynamicCallback (386 ms)
[ RUN      ] TestAMQP.ReceiveNack
[       OK ] TestAMQP.ReceiveNack (369 ms)
[ RUN      ] TestAMQP.FailWrite
[       OK ] TestAMQP.FailWrite (318 ms)
[ RUN      ] TestAMQP.ClosedConnection
[       OK ] TestAMQP.ClosedConnection (634 ms)
[ RUN      ] TestAMQP.RetryInvalidHost
[       OK ] TestAMQP.RetryInvalidHost (344 ms)
[ RUN      ] TestAMQP.RetryInvalidPort
[       OK ] TestAMQP.RetryInvalidPort (322 ms)
[ RUN      ] TestAMQP.RetryFailWrite
[       OK ] TestAMQP.RetryFailWrite (1231 ms)
[ RUN      ] TestAMQP.AcksWithReconnect
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:466: Failure
Expected equality of these values:
  callbacks_invoked
    Which is: 198
  2*NUMBER_OF_CALLS
    Which is: 200
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rgw/test_rgw_amqp.cc:424: Failure
Expected equality of these values:
  rc
    Which is: -4098
  0
[  FAILED  ] TestAMQP.AcksWithReconnect (607 ms)
[----------] 21 tests from TestAMQP (6184 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 1 test suite ran. (6184 ms total)
[  PASSED  ] 18 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] TestAMQP.ReceiveMultipleAck
[  FAILED  ] TestAMQP.ReceiveAckForMultiple
[  FAILED  ] TestAMQP.AcksWithReconnect

 3 FAILED TESTS

i started seeing the failures since last week.

https://jenkins.ceph.com/job/ceph-pull-requests/34689/console. the link will return 404 in couple days.


Related issues

Duplicated by rgw - Bug #39986: unittest_rgw_amqp failing transiently in nautilus PRs Duplicate 05/21/2019
Copied to rgw - Backport #42277: nautilus: unittest_rgw_amqp failing in master Resolved

History

#1 Updated by Kefu Chai 3 months ago

  • Related to Bug #39986: unittest_rgw_amqp failing transiently in nautilus PRs added

#2 Updated by Kefu Chai 2 months ago

  • Priority changed from Normal to High

mark this ticket high. as i found this test failed more and more frequently.

#3 Updated by Yuval Lifshitz 2 months ago

  • Pull request ID set to 30735

addressed by PR: https://github.com/ceph/ceph/pull/30735
once merged to master, same commit will be backmerged to nautilus to fix: https://tracker.ceph.com/issues/39986
current amqp code is similar between master and nautilus

#4 Updated by Kefu Chai 2 months ago

  • Status changed from New to Fix Under Review

#5 Updated by Nathan Cutler 2 months ago

  • Backport set to nautilus

Adding nautilus backport here and will close the nautilus-specific tracker as duplicate of this one.

#6 Updated by Nathan Cutler 2 months ago

  • Related to deleted (Bug #39986: unittest_rgw_amqp failing transiently in nautilus PRs)

#7 Updated by Nathan Cutler 2 months ago

  • Duplicated by Bug #39986: unittest_rgw_amqp failing transiently in nautilus PRs added

#8 Updated by Nathan Cutler 2 months ago

@Yuval - when the master PR merges, please change the status of this issue to "Pending Backport".

#9 Updated by Kefu Chai 2 months ago

  • Status changed from Fix Under Review to Resolved

#10 Updated by Kefu Chai 2 months ago

  • Status changed from Resolved to Pending Backport

#11 Updated by Nathan Cutler 2 months ago

  • Copied to Backport #42277: nautilus: unittest_rgw_amqp failing in master added

#12 Updated by Nathan Cutler about 1 month ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF