Project

General

Profile

Bug #58173

api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolEIOFlag

Added by Adam King about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Urgent
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The workunits/rados/test.sh script is run in the orch suite on some tests. In a few of them, these two tests were failing.
The run did include some unmerged PRs but they all only touched code within mgr modules so considering it unlikely they caused the failures.
Additionally, it was inconsistent whether one, the other, or both tests referenced in the tracker title failed.

https://pulpito.ceph.com/adking-2022-12-02_21:56:44-orch:cephadm-wip-adk-testing-2022-12-02-0932-distro-default-smithi/7101924/

2022-12-03T02:55:13.492 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.SimplePoolEIOFlag
2022-12-03T02:55:13.493 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: [       OK ] LibRadosAio.SimplePoolEIOFlag (3986 ms)
2022-12-03T02:55:13.493 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.PoolEIOFlag
2022-12-03T02:55:13.493 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: setting pool EIO
2022-12-03T02:55:13.494 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: max_success 100, min_failed 0
2022-12-03T02:55:13.494 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1172-g20d17790/rpm/el8/BUILD/ceph-18.0.0-1172-g20d17790/src/test/librados/aio_cxx.cc:2376: Failure
2022-12-03T02:55:13.494 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: Value of: max_success + 1 == min_failed
2022-12-03T02:55:13.495 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp:   Actual: false
2022-12-03T02:55:13.495 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: Expected: true
2022-12-03T02:55:13.495 INFO:tasks.workunit.client.0.smithi120.stdout:               api_aio_pp: [  FAILED  ] LibRadosAio.PoolEIOFlag (4017 ms)

https://pulpito.ceph.com/adking-2022-12-02_21:56:44-orch:cephadm-wip-adk-testing-2022-12-02-0932-distro-default-smithi/7101936/

2022-12-03T03:12:42.349 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.SimplePoolEIOFlag
2022-12-03T03:12:42.350 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: /build/ceph-18.0.0-1172-g20d17790/src/test/librados/aio_cxx.cc:2317: Failure
2022-12-03T03:12:42.350 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: Expected equality of these values:
2022-12-03T03:12:42.350 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp:   -5
2022-12-03T03:12:42.350 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp:   my_completion->get_return_value()
2022-12-03T03:12:42.351 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp:     Which is: 0
2022-12-03T03:12:42.351 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: [  FAILED  ] LibRadosAio.SimplePoolEIOFlag (3574 ms)
2022-12-03T03:12:42.351 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.PoolEIOFlag
2022-12-03T03:12:42.351 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: setting pool EIO
2022-12-03T03:12:42.352 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: waiting for 99,100
2022-12-03T03:12:42.352 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: max_success 100, min_failed 0
2022-12-03T03:12:42.352 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: /build/ceph-18.0.0-1172-g20d17790/src/test/librados/aio_cxx.cc:2376: Failure
2022-12-03T03:12:42.352 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: Value of: max_success + 1 == min_failed
2022-12-03T03:12:42.353 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp:   Actual: false
2022-12-03T03:12:42.353 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: Expected: true
2022-12-03T03:12:42.353 INFO:tasks.workunit.client.0.smithi090.stdout:               api_aio_pp: [  FAILED  ] LibRadosAio.PoolEIOFlag (5013 ms)

https://pulpito.ceph.com/adking-2022-12-02_21:56:44-orch:cephadm-wip-adk-testing-2022-12-02-0932-distro-default-smithi/7101923/

2022-12-03T02:56:59.081 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.SimplePoolEIOFlag
2022-12-03T02:56:59.082 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1172-g20d17790/rpm/el8/BUILD/ceph-18.0.0-1172-g20d17790/src/test/librados/aio_cxx.cc:2317: Failure
2022-12-03T02:56:59.082 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: Expected equality of these values:
2022-12-03T02:56:59.082 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp:   -5
2022-12-03T02:56:59.082 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp:   my_completion->get_return_value()
2022-12-03T02:56:59.083 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp:     Which is: 0
2022-12-03T02:56:59.083 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: [  FAILED  ] LibRadosAio.SimplePoolEIOFlag (3563 ms)
2022-12-03T02:56:59.083 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.PoolEIOFlag
2022-12-03T02:56:59.083 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: setting pool EIO
2022-12-03T02:56:59.084 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: max_success 95, min_failed 96
2022-12-03T02:56:59.084 INFO:tasks.workunit.client.0.smithi047.stdout:               api_aio_pp: [       OK ] LibRadosAio.PoolEIOFlag (4018 ms)


Related issues

Duplicated by crimson - Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag Duplicate

History

#1 Updated by Adam King about 2 months ago

  • Project changed from Ceph to RADOS

#2 Updated by Neha Ojha about 2 months ago

  • Priority changed from Normal to Urgent

#3 Updated by Samuel Just about 2 months ago

  • Related to Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag added

#4 Updated by Samuel Just about 2 months ago

  • Related to deleted (Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag)

#5 Updated by Samuel Just about 2 months ago

  • Duplicated by Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag added

#6 Updated by Samuel Just about 2 months ago

Matan added that test within the last two weeks: https://github.com/ceph/ceph/pull/49029

#7 Updated by Laura Flores about 2 months ago

  • Related to Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag added

#8 Updated by Laura Flores about 2 months ago

  • Related to deleted (Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag)

#9 Updated by Laura Flores about 2 months ago

Excuse my update Sam, I see you already added it as a duplicate.

#10 Updated by Laura Flores about 2 months ago

Building a branch here with https://github.com/ceph/ceph/pull/49029 reverted, which can be used to verify whether it is breaking tests.

https://shaman.ceph.com/builds/ceph/wip-pr-49029-reverted/6c5f7b3b7baab53895e2a3b6f8b6802851cc5cc3/

#11 Updated by Matan Breizman about 2 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Matan Breizman
  • Pull request ID set to 49278

#12 Updated by Laura Flores about 2 months ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF