Project

General

Profile

Bug #19741

test_envlibrados_for_rocksdb.yaml fails on crypto restart

Added by Sage Weil 6 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
Start date:
04/21/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

2017-04-21T06:12:51.987 INFO:tasks.workunit.client.0.smithi033.stdout:[----------] 4 tests from EnvLibradosMutipoolTest
2017-04-21T06:12:51.987 INFO:tasks.workunit.client.0.smithi033.stdout:[ RUN      ] EnvLibradosMutipoolTest.Basics
2017-04-21T06:12:56.223 INFO:tasks.workunit.client.0.smithi033.stdout:[       OK ] EnvLibradosMutipoolTest.Basics (4236 ms)
2017-04-21T06:12:56.223 INFO:tasks.workunit.client.0.smithi033.stdout:[ RUN      ] EnvLibradosMutipoolTest.DBBasics
2017-04-21T06:13:01.518 INFO:tasks.workunit.client.0.smithi033.stdout:[       OK ] EnvLibradosMutipoolTest.DBBasics (5295 ms)
2017-04-21T06:13:01.519 INFO:tasks.workunit.client.0.smithi033.stdout:[ RUN      ] EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder
2017-04-21T06:13:03.733 INFO:tasks.workunit.client.0.smithi033.stdout:Test size : loop(64); bulk_size(32768)
2017-04-21T06:13:25.932 INFO:tasks.workunit.client.0.smithi033.stdout:Time by default : 7778ms
2017-04-21T06:13:35.237 INFO:tasks.workunit.client.0.smithi033.stdout:Time by librados : 9241ms
2017-04-21T06:13:35.248 INFO:tasks.workunit.client.0.smithi033.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.1-1252-gfc32356/rpm/el7/BUILD/ceph-12.0.1-1252-gfc32356/src/common/ceph_crypto.cc: In func
tion 'void ceph::crypto::init(CephContext*)' thread 7efdd7c87700 time 2017-04-21 06:13:35.245797
2017-04-21T06:13:35.248 INFO:tasks.workunit.client.0.smithi033.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.1-1252-gfc32356/rpm/el7/BUILD/ceph-12.0.1-1252-gfc32356/src/common/ceph_crypto.cc: 77: FAI
LED assert(crypto_context != __null)
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: ceph version 12.0.1-1252-gfc32356 (fc3235649273967d76678eb3723e7701867c5707)
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7efdcd651dd0]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 2: (ceph::crypto::init(CephContext*)+0x103) [0x7efdcd866953]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 3: (CephContext::init_crypto()+0x15) [0x7efdcd811805]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 4: (common_init_finish(CephContext*)+0x10) [0x7efdcd80e320]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 5: (librados::RadosClient::connect()+0x30) [0x7efdd77d73b0]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 6: ./env_librados_test() [0x4f8909]
2017-04-21T06:13:35.249 INFO:tasks.workunit.client.0.smithi033.stderr: 7: ./env_librados_test() [0x4f8da8]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 8: ./env_librados_test() [0x7ad729]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 9: ./env_librados_test() [0x7a119a]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 10: ./env_librados_test() [0x7a1297]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 11: ./env_librados_test() [0x7a151d]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 12: ./env_librados_test() [0x7a17f3]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 13: ./env_librados_test() [0x410de1]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 14: (__libc_start_main()+0xf5) [0x7efdd5f02b35]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: 15: ./env_librados_test() [0x4e3aca]
2017-04-21T06:13:35.250 INFO:tasks.workunit.client.0.smithi033.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

this is likely related to https://github.com/ceph/ceph/pull/12624 and #14115 in that there is a crypto key being leaked, preventing nss shutdown, which is making it fail to restart.

/a/sage-2017-04-21_02:10:47-rados-wip-sage-testing2---basic-smithi/1050955
but this has been failing off an on forever.


Related issues

Duplicated by Ceph - Bug #18478: "FAILED assert(crypto_context != __null)" in rados-kraken-distro-basic-smithi New 01/10/2017
Copied to Ceph - Backport #20460: jewel: test_envlibrados_for_rocksdb.yaml fails on crypto restart Resolved
Copied to Ceph - Backport #20461: kraken: test_envlibrados_for_rocksdb.yaml fails on crypto restart Rejected

History

#1 Updated by Sage Weil 6 months ago

/a/sage-2017-04-28_00:12:30-rados-wip-sage-testing2---basic-smithi/1075455

#2 Updated by Kefu Chai 6 months ago

/a/kchai-2017-05-06_17:11:36-rados-wip-kefu-testing---basic-mira/1109228/teuthology.log

#3 Updated by Brad Hubbard 5 months ago

/a/bhubbard-2017-05-24_05:25:43-rados-wip-badone-testing---basic-smithi/1224665/teuthology.log

#4 Updated by Sage Weil 5 months ago

/a/sage-2017-05-26_06:56:21-rados-wip-sage-testing---basic-smithi/1231184 (centos)

#5 Updated by Sage Weil 5 months ago

/a/sage-2017-06-02_08:32:01-rados-wip-sage-testing-distro-basic-smithi/1255399

#6 Updated by Kefu Chai 4 months ago

an AsyncConnection gets leaked because a Message (an MOSDOpReply in this case) holding a reference to it is leaked.

#7 Updated by Kefu Chai 4 months ago

  • Assignee set to Kefu Chai

#8 Updated by Kefu Chai 4 months ago

  • Related to Bug #18478: "FAILED assert(crypto_context != __null)" in rados-kraken-distro-basic-smithi added

#9 Updated by Kefu Chai 4 months ago

  • Status changed from Verified to Resolved

#10 Updated by Kefu Chai 4 months ago

  • Related to deleted (Bug #18478: "FAILED assert(crypto_context != __null)" in rados-kraken-distro-basic-smithi)

#11 Updated by Kefu Chai 4 months ago

  • Duplicated by Bug #18478: "FAILED assert(crypto_context != __null)" in rados-kraken-distro-basic-smithi added

#12 Updated by Kefu Chai 4 months ago

  • Status changed from Resolved to Pending Backport
  • Backport set to jewel, kraken

#13 Updated by Kefu Chai 4 months ago

  • Assignee deleted (Kefu Chai)

#15 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #20460: jewel: test_envlibrados_for_rocksdb.yaml fails on crypto restart added

#16 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #20461: kraken: test_envlibrados_for_rocksdb.yaml fails on crypto restart added

#18 Updated by Nathan Cutler about 2 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF