Project

General

Profile

Actions
Copy link

Bug #23517

closed

TestMockDeepCopyRequest.SimpleCopy fails in run-rbd-unit-tests.sh

Added by Kefu Chai about 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Tests
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[ RUN      ] TestMockDeepCopyRequest.SimpleCopy
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_mock_DeepCopyRequest.cc:152: Failure
      Expected: 0
To be equal to: open_image(m_image_name, &m_src_image_ctx)
      Which is: -38

for the first time i found this failure in amd64's run of "make check", this test has been failing on arch64 for couple weeks.

see https://jenkins.ceph.com/job/ceph-pull-requests/43343/consoleFull#1862740416733401c-e9d0-4737-9832-6594c5da0afa . this link will be invalid very soon thou.

Actions
Copy link
 #1

Updated by Jason Dillaman about 6 years ago

Moving this to CI. This failure would only occur if the cls_XYX.so libraries could not be loaded during the execution of the unit test (notice all the other -ENOSYS failures as well). Perhaps it's one or two builders that are out-of-whack with something causing the CEPH_LIB environment variable to point to the wrong place.

Actions
Copy link
 #2

Updated by Kefu Chai about 6 years ago

  • Project changed from rbd to RADOS
  • Status changed from New to 12
  • Assignee set to Kefu Chai

not able to move this to CI somehow... moving it to RADOS.

Actions
Copy link
 #3

Updated by Kefu Chai about 6 years ago

  • Category set to Tests
  • Status changed from 12 to Fix Under Review
  • Source set to Development
+ ls -l /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib
+ unset RBD_FEATURES
+ unittest_librbd
lrwxrwxrwx. 1 jenkins-build jenkins-build        17 Apr 19 06:00 libceph_zlib.so -> libceph_zlib.so.2
lrwxrwxrwx. 1 jenkins-build jenkins-build        21 Apr 19 06:00 libceph_zlib.so.2 -> libceph_zlib.so.2.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   2534504 Apr 19 06:00 libceph_zlib.so.2.0.0
lrwxrwxrwx. 1 jenkins-build jenkins-build        17 Apr 19 05:59 libceph_zstd.so -> libceph_zstd.so.2
lrwxrwxrwx. 1 jenkins-build jenkins-build        21 Apr 19 05:59 libceph_zstd.so.2 -> libceph_zstd.so.2.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1171408 Apr 19 05:59 libceph_zstd.so.2.0.0
lrwxrwxrwx. 1 jenkins-build jenkins-build        14 Apr 19 06:11 libcephfs.so -> libcephfs.so.2
lrwxrwxrwx. 1 jenkins-build jenkins-build        18 Apr 19 06:11 libcephfs.so.2 -> libcephfs.so.2.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build  32789440 Apr 19 06:11 libcephfs.so.2.0.0
-rw-r--r--. 1 jenkins-build jenkins-build  78303240 Apr 19 06:09 libclient.a
open_all_classes CEPH_LIB = /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib
open_all_classes opening: libcls_lock.so
open_class loaded class: lock
open_all_classes opening: libcls_version.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_version.so: undefined symbol: _Z19cls_gen_rand_base64Pci
open_all_classes opening: libcls_journal.so
open_class loaded class: journal
open_all_classes opening: libcls_statelog.so
open_class loaded class: statelog
open_all_classes opening: libcls_log.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_log.so: undefined symbol: _Z23cls_cxx_map_read_headerPvPN4ceph6buffer4listE
open_all_classes opening: libcls_timeindex.so
open_class loaded class: timeindex
open_all_classes opening: libcls_replica_log.so
open_class loaded class: replica_log
open_all_classes opening: libcls_user.so
lrwxrwxrwx. 1 jenkins-build jenkins-build        19 Apr 19 06:00 libcls_journal.so -> libcls_journal.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        23 Apr 19 06:00 libcls_journal.so.1 -> libcls_journal.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1794576 Apr 19 06:00 libcls_journal.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   4415596 Apr 19 06:00 libcls_journal_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        16 Apr 19 05:59 libcls_lock.so -> libcls_lock.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        20 Apr 19 05:59 libcls_lock.so.1 -> libcls_lock.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1431464 Apr 19 05:59 libcls_lock.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   5840252 Apr 19 05:59 libcls_lock_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        15 Apr 19 06:00 libcls_log.so -> libcls_log.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        19 Apr 19 06:00 libcls_log.so.1 -> libcls_log.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1084504 Apr 19 06:00 libcls_log.so.1.0.0
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_user.so: undefined symbol: _Z23cls_cxx_map_read_headerPvPN4ceph6buffer4listE
open_all_classes opening: libcls_rbd.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_user.so: undefined symbol: _Z23cls_cxx_map_read_headerPvPN4ceph6buffer4listE
open_all_classes opening: libcls_refcount.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_refcount.so: undefined symbol: _Z14cls_cxx_removePv
open_all_classes opening: libcls_rgw.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_rgw.so: undefined symbol: _Z23cls_cxx_map_read_headerPvPN4ceph6buffer4listE
get_method no such a cls rbd
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/run-rbd-unit-tests.sh: line 12:  3540 Segmentation fault      unittest_librbd
-rw-r--r--. 1 jenkins-build jenkins-build   1861726 Apr 19 05:59 libcls_log_client.a
-rw-r--r--. 1 jenkins-build jenkins-build   6300838 Apr 19 06:00 libcls_otp_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        15 Apr 19 06:00 libcls_rbd.so -> libcls_rbd.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        19 Apr 19 06:00 libcls_rbd.so.1 -> libcls_rbd.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   6417904 Apr 19 06:00 libcls_rbd.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   8360316 Apr 19 06:00 libcls_rbd_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        20 Apr 19 06:02 libcls_refcount.so -> libcls_refcount.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        24 Apr 19 06:02 libcls_refcount.so.1 -> libcls_refcount.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build  20296120 Apr 19 06:02 libcls_refcount.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   2817370 Apr 19 05:59 libcls_refcount_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        23 Apr 19 06:00 libcls_replica_log.so -> libcls_replica_log.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        27 Apr 19 06:00 libcls_replica_log.so.1 -> libcls_replica_log.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1038408 Apr 19 06:00 libcls_replica_log.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   5162274 Apr 19 05:59 libcls_replica_log_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        15 Apr 19 06:03 libcls_rgw.so -> libcls_rgw.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        19 Apr 19 06:03 libcls_rgw.so.1 -> libcls_rgw.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build  26180168 Apr 19 06:03 libcls_rgw.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build  17853332 Apr 19 06:01 libcls_rgw_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        20 Apr 19 06:00 libcls_statelog.so -> libcls_statelog.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        24 Apr 19 06:00 libcls_statelog.so.1 -> libcls_statelog.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1210392 Apr 19 06:00 libcls_statelog.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   1913872 Apr 19 05:59 libcls_statelog_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        21 Apr 19 06:00 libcls_timeindex.so -> libcls_timeindex.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        25 Apr 19 06:00 libcls_timeindex.so.1 -> libcls_timeindex.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build    985944 Apr 19 06:00 libcls_timeindex.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   1655516 Apr 19 05:59 libcls_timeindex_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        16 Apr 19 06:00 libcls_user.so -> libcls_user.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        20 Apr 19 06:00 libcls_user.so.1 -> libcls_user.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build   1436208 Apr 19 06:00 libcls_user.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   6049738 Apr 19 06:00 libcls_user_client.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        19 Apr 19 06:00 libcls_version.so -> libcls_version.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        23 Apr 19 06:00 libcls_version.so.1 -> libcls_version.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build    832144 Apr 19 06:00 libcls_version.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   2148740 Apr 19 05:59 libcls_version_client.a
-rw-r--r--. 1 jenkins-build jenkins-build 496901124 Apr 19 06:07 libcommon.a
-rw-r--r--. 1 jenkins-build jenkins-build      8788 Apr 19 05:55 libcommon_crc_aarch64.a
-rw-r--r--. 1 jenkins-build jenkins-build     11960 Apr 19 05:55 libcommon_utf8.a
-rw-r--r--. 1 jenkins-build jenkins-build    258002 Apr 19 05:59 libdmclock.a
-rwxr-xr-x. 1 jenkins-build jenkins-build   3851088 Apr 19 06:00 libec_example.so
-rwxr-xr-x. 1 jenkins-build jenkins-build     71776 Apr 19 05:59 libec_fail_to_initialize.so
-rwxr-xr-x. 1 jenkins-build jenkins-build     71656 Apr 19 05:59 libec_fail_to_register.so
-rwxr-xr-x. 1 jenkins-build jenkins-build     72104 Apr 19 05:59 libec_hangs.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   5625880 Apr 19 06:00 libec_jerasure.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   5625864 Apr 19 06:00 libec_jerasure_generic.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   5625864 Apr 19 06:00 libec_jerasure_neon.so
-rwxr-xr-x. 1 jenkins-build jenkins-build  34201648 Apr 19 06:03 libec_lrc.so
-rwxr-xr-x. 1 jenkins-build jenkins-build     71280 Apr 19 05:59 libec_missing_entry_point.so
-rwxr-xr-x. 1 jenkins-build jenkins-build     71072 Apr 19 05:59 libec_missing_version.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   6661056 Apr 19 06:01 libec_shec.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   6661048 Apr 19 06:00 libec_shec_generic.so
-rwxr-xr-x. 1 jenkins-build jenkins-build   6661048 Apr 19 06:00 libec_shec_neon.so
-rw-r--r--. 1 jenkins-build jenkins-build   1268472 Apr 19 05:59 liberasure_code.a
-rw-r--r--. 1 jenkins-build jenkins-build   7784086 Apr 19 06:09 libglobal-static.a
-rw-r--r--. 1 jenkins-build jenkins-build   7784086 Apr 19 06:09 libglobal.a
-rw-r--r--. 1 jenkins-build jenkins-build   8542164 Apr 19 05:59 libgmock.a
-rw-r--r--. 1 jenkins-build jenkins-build   8868296 Apr 19 05:59 libgmock_main.a
-rw-r--r--. 1 jenkins-build jenkins-build   5943814 Apr 19 05:59 libgtest.a
-rw-r--r--. 1 jenkins-build jenkins-build  32083846 Apr 19 06:01 libjournal.a
-rw-r--r--. 1 jenkins-build jenkins-build   2915914 Apr 19 05:59 libjournal_test_mock.a
-rw-r--r--. 1 jenkins-build jenkins-build  27721884 Apr 19 06:01 libjson_spirit.a
-rw-r--r--. 1 jenkins-build jenkins-build   4004288 Apr 19 05:59 libkrbd.a
-rw-r--r--. 1 jenkins-build jenkins-build  20262706 Apr 19 06:01 libkv.a
-rw-r--r--. 1 jenkins-build jenkins-build   1878830 Apr 19 05:55 liblua.a
-rw-r--r--. 1 jenkins-build jenkins-build 629339646 Apr 19 06:12 libmds.a
-rw-r--r--. 1 jenkins-build jenkins-build 308182822 Apr 19 06:09 libmon.a
-rw-r--r--. 1 jenkins-build jenkins-build 208230972 Apr 19 06:09 libos.a
-rw-r--r--. 1 jenkins-build jenkins-build 401729856 Apr 19 06:10 libosd.a
-rw-r--r--. 1 jenkins-build jenkins-build  44861842 Apr 19 06:02 libosdc.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        13 Apr 19 06:09 librados.so -> librados.so.2
lrwxrwxrwx. 1 jenkins-build jenkins-build        17 Apr 19 06:09 librados.so.2 -> librados.so.2.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build  26407576 Apr 19 06:09 librados.so.2.0.0
-rw-r--r--. 1 jenkins-build jenkins-build  36103900 Apr 19 06:01 librados_test_stub.a
-rw-r--r--. 1 jenkins-build jenkins-build  57160384 Apr 19 06:23 libradosgw_a.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        20 Apr 19 06:13 libradosstriper.so -> libradosstriper.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        24 Apr 19 06:13 libradosstriper.so.1 -> libradosstriper.so.1.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build  23893648 Apr 19 06:13 libradosstriper.so.1.0.0
-rw-r--r--. 1 jenkins-build jenkins-build   7787812 Apr 19 06:11 libradostest.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        11 Apr 19 06:13 librbd.so -> librbd.so.1
lrwxrwxrwx. 1 jenkins-build jenkins-build        16 Apr 19 06:13 librbd.so.1 -> librbd.so.1.12.0
-rwxr-xr-x. 1 jenkins-build jenkins-build 149796096 Apr 19 06:13 librbd.so.1.12.0
-rw-r--r--. 1 jenkins-build jenkins-build  10073986 Apr 19 06:00 librbd_api.a
-rw-r--r--. 1 jenkins-build jenkins-build 454465776 Apr 19 06:08 librbd_internal.a
-rw-r--r--. 1 jenkins-build jenkins-build 146386864 Apr 19 06:04 librbd_mirror_internal.a
-rw-r--r--. 1 jenkins-build jenkins-build  47555972 Apr 19 06:02 librbd_mirror_test.a
-rw-r--r--. 1 jenkins-build jenkins-build   5928198 Apr 19 06:00 librbd_mirror_types.a
-rw-r--r--. 1 jenkins-build jenkins-build  12019108 Apr 19 06:15 librbd_replay.a
-rw-r--r--. 1 jenkins-build jenkins-build   2785656 Apr 19 06:14 librbd_replay_ios.a
-rw-r--r--. 1 jenkins-build jenkins-build   4647626 Apr 19 05:59 librbd_replay_types.a
-rw-r--r--. 1 jenkins-build jenkins-build  98427644 Apr 19 06:05 librbd_test.a
-rw-r--r--. 1 jenkins-build jenkins-build   7423608 Apr 19 06:00 librbd_test_mock.a
-rw-r--r--. 1 jenkins-build jenkins-build  22828042 Apr 19 06:01 librbd_types.a
lrwxrwxrwx. 1 jenkins-build jenkins-build        11 Apr 19 06:22 librgw.so -> librgw.so.2
lrwxrwxrwx. 1 jenkins-build jenkins-build        15 Apr 19 06:22 librgw.so.2 -> librgw.so.2.0.0
-rwxr-xr-x. 1 jenkins-build jenkins-build 171394288 Apr 19 06:22 librgw.so.2.0.0
-rw-r--r--. 1 jenkins-build jenkins-build 667191742 Apr 19 06:20 librgw_a.a
seed 3540
[==========] Running 659 tests from 66 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from TestMockDeepCopyRequest
[ RUN      ] TestMockDeepCopyRequest.SimpleCopy
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_mock_DeepCopyRequest.cc:152: Failure
      Expected: 0
To be equal to: open_image(m_image_name, &m_src_image_ctx)
      Which is: -38

i added more verbose log when loading an object class. and the log is posted above. the funny thing is that when libcls_rbd fails to load, the error message referenced to libcls_user:

open_all_classes opening: libcls_rbd.so
open_class Failed to load class: /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/lib/libcls_user.so: undefined symbol: _Z23cls_cxx_map_read_headerPvPN4ceph6buffer4listE

and ldd shows that libcls_rbd was not referencing libcls_user

[root@omani006 lib]# ldd libcls_rbd.so
    linux-vdso.so.1 =>  (0x000003ffb4310000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x000003ffb4140000)
    libm.so.6 => /lib64/libm.so.6 (0x000003ffb4080000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000003ffb4040000)
    libc.so.6 => /lib64/libc.so.6 (0x000003ffb3ec0000)
    /lib/ld-linux-aarch64.so.1 (0x000002aabcf40000)

and neither is it referencing the cxx_map_read_header symbol

[root@omani006 lib]# nm -C libcls_rbd.so  | grep cxx_map_read_header

anyway, to appease dlopen(), i am adding object containing missing symbol to the executable. see

https://github.com/ceph/ceph/pull/21280

Actions
Copy link
 #5

Updated by Kefu Chai about 6 years ago

  • Status changed from Fix Under Review to 12
  • Assignee deleted (Kefu Chai)
Actions
Copy link
 #7

Updated by Jason Dillaman about 6 years ago

  • Status changed from 12 to Resolved
Actions
Copy link

Also available in: Atom PDF