Bug #44582: LibRadosMisc.ShutdownRace - RADOS - Ceph

Actions

Copy link

Bug #44582

closed

LibRadosMisc.ShutdownRace

Added by Sage Weil about 4 years ago. Updated about 4 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

34011

Crash signature (v1):

Crash signature (v2):

Description

2020-03-12T13:22:33.141 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: [ RUN      ] LibRadosMisc.ShutdownRace
2020-03-12T13:22:33.141 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/MACHINE_TYPE/braggi/release/15.1.0-2073-ge7b4f03/rpm/el8/BUILD/ceph-15.1.0-2073-ge7b4f03/src/test/librados/misc.cc:321: Failure
2020-03-12T13:22:33.141 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: Expected equality of these values:
2020-03-12T13:22:33.141 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   "" 
2020-03-12T13:22:33.141 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/MACHINE_TYPE/braggi/release/15.1.0-2073-ge7b4f03/rpm/el8/BUILD/ceph-15.1.0-2073-ge7b4f03/src/test/librados/misc.cc:321: Failure
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: Expected equality of these values:
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   "" 
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
2020-03-12T13:22:33.142 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/MACHINE_TYPE/braggi/release/15.1.0-2073-ge7b4f03/rpm/el8/BUILD/ceph-15.1.0-2073-ge7b4f03/src/test/librados/misc.cc:321: Failure
...
2020-03-12T13:27:11.742 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: Expected equality of these values:
2020-03-12T13:27:11.742 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   "" 
2020-03-12T13:27:11.742 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-12T13:27:11.742 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
2020-03-12T13:27:11.742 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/MACHINE_TYPE/braggi/release/15.1.0-2073-ge7b4f03/rpm/el8/BUILD/ceph-15.1.0-2073-ge7b4f03/src/test/librados/misc.cc:321: Failure
2020-03-12T13:27:11.743 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: Expected equality of these values:
2020-03-12T13:27:11.743 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   "" 
2020-03-12T13:27:11.743 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-12T13:27:11.743 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
2020-03-12T13:27:11.743 INFO:tasks.workunit.client.0.smithi088.stdout:                 api_misc: [  FAILED  ] LibRadosMisc.ShutdownRace (581851 ms)

/a/sage-2020-03-12_03:40:31-rados-wip-sage2-testing-2020-03-11-1701-distro-basic-smithi/4849123

Actions

Copy link

Updated by Sage Weil about 4 years ago

/a/sage-2020-03-17_14:06:37-rados:verify-wip-sage-testing-2020-03-16-2107-distro-basic-smithi/4863370

Actions

Copy link

Updated by Sage Weil about 4 years ago

110 is ETIMEDOUT

2020-03-17T15:16:07.728 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc: [ RUN      ] LibRadosMisc.ShutdownRace
2020-03-17T15:16:07.728 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/MACHINE_TYPE/braggi/release/15.1.1-89-g5763e93/rpm/el8/BUILD/ceph-15.1.1-89-g5763e93/src/test/librados/misc.cc:321: Failure
2020-03-17T15:16:07.728 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc: Expected equality of these values:
2020-03-17T15:16:07.729 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:   "" 
2020-03-17T15:16:07.729 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-17T15:16:07.729 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
...
2020-03-17T15:25:20.038 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:   "" 
2020-03-17T15:25:20.038 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:   connect_cluster(&rad)
2020-03-17T15:25:20.039 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc:     Which is: "rados_connect failed with error -110" 
2020-03-17T15:25:20.039 INFO:tasks.workunit.client.0.smithi103.stdout:                 api_misc: [  FAILED  ] LibRadosMisc.ShutdownRace (854728 ms)

that's like 15 minutes! the test is doing 128 attempts

Actions

Copy link