Bug #39995
closedrgw: make check is failing because of linker error on Ubuntu
0%
Description
It happens only on Ubuntu, CentOS is fine:
- https://jenkins.ceph.com/job/ceph-pull-requests/359/consoleFull
- https://jenkins.ceph.com/job/ceph-pull-requests/490/consoleFull
/usr/bin/ld: warning: libboost_context.so.1.67.0, needed by //opt/ceph/lib/x86_64-linux-gnu/libboost_coroutine.so.1.67.0, not found (try using -rpath or -rpath-link) //opt/ceph/lib/x86_64-linux-gnu/libboost_coroutine.so.1.67.0: undefined reference to `jump_fcontext' //opt/ceph/lib/x86_64-linux-gnu/libboost_coroutine.so.1.67.0: undefined reference to `make_fcontext' collect2: error: ld returned 1 exit status Link Error: RGW library not found LINK test/unit/lib/iscsi/init_grp.c/init_grp_ut src/pybind/rgw/CMakeFiles/cython_rgw.dir/build.make:57: recipe for target 'src/pybind/rgw/CMakeFiles/cython_rgw' failed make[3]: *** [src/pybind/rgw/CMakeFiles/cython_rgw] Error 1 CMakeFiles/Makefile2:5456: recipe for target 'src/pybind/rgw/CMakeFiles/cython_rgw.dir/all' failed make[2]: *** [src/pybind/rgw/CMakeFiles/cython_rgw.dir/all] Error 2
Updated by Sebastian Wagner almost 5 years ago
- Status changed from New to 12
- Priority changed from Normal to Urgent
Updated by Nathan Cutler almost 5 years ago
- Related to Bug #39981: do_cmake.sh: "source" not found added
Updated by Nathan Cutler almost 5 years ago
Maybe this is the root cause? #39981
Updated by Nathan Cutler almost 5 years ago
All of the console logs are from a single PR: https://github.com/ceph/ceph/pull/19371
And "make check" just passed in that PR following merge of https://github.com/ceph/ceph/pull/28181 which is. . . a positive sign.
Updated by Sebastian Wagner almost 5 years ago
Updated by Nathan Cutler almost 5 years ago
Still happening: https://github.com/ceph/ceph/pull/28211
So I'm removing the related-to link to #39981.
Updated by Nathan Cutler almost 5 years ago
- Related to deleted (Bug #39981: do_cmake.sh: "source" not found)
Updated by Sebastian Wagner almost 5 years ago
Updated by Sebastian Wagner almost 5 years ago
- Assignee set to Casey Bodley
(Ceasy, assigned to you, just to make you aware of this.)
Updated by Sebastian Wagner almost 5 years ago
Updated by Sebastian Wagner almost 5 years ago
Updated by Casey Bodley almost 5 years ago
I set up a fresh bionic vm and was able to build master successfully. The ubuntu flavors of master and nautilus are still succeeding in https://shaman.ceph.com/builds/ceph/. It looks like the difference here is the use of WITH_SYSTEM_BOOST and the ceph-libboost-* packages.
From the logs of failures, I do see that cmake is able to find the boost context component:
-- Boost version: 1.67.0 -- Found the following Boost libraries: -- atomic -- chrono -- thread -- system -- regex -- random -- program_options -- date_time -- iostreams -- python27 -- context -- coroutine -- filesystem -- timer
But the failure implies that libboost_context.so isn't present under /opt/ceph/:
/usr/bin/ld: warning: libboost_context.so.1.67.0, needed by //opt/ceph/lib/x86_64-linux-gnu/libboost_coroutine.so.1.67.0, not found (try using -rpath or -rpath-link)
Updated by Casey Bodley almost 5 years ago
The logs of failures seem to be using xenial:
++ VERSION='16.04.4 LTS (Xenial Xerus)'
So I created a xenial vm and again built successfully with:
cmake -DWITH_SYSTEM_BOOST=ON -DBOOST_ROOT=/opt/ceph ..
Updated by Luis Henriques almost 5 years ago
Maybe just a shot in the dark, but can't the problem be in the libboost-* libs being installed? The "undefined reference" error occurs in libboost_coroutine.so.1.67.0, which is installed from some https://chacra.ceph.com/repos/... URL. Are these libs sane, or could they be already broken?
Updated by Kefu Chai almost 5 years ago
td;dr ceph-libboost1.67-dev_1.67.0-7_amd64.deb was not sane.
long story,
- before https://github.com/tchaikov/ceph-boost/commit/ec205d3f54258c0d7f9d22ae279b1a20523e7629 was merged, we could consistently reproduce this issue.
- I tried to fix it by rebuilding the ceph-libboost* packages and uploading them to chacra. but i failed to reinstall them to all xenial builders.
- and re-running install-deps.sh cannot help because:
- i was using the same changelog file when rebuilding the ceph-libboost* packages. as i thought it would be simpler to fix the builders than updating install-deps.sh, if the ceph-libboost* packages went wrong. so the new ceph-libboost* packages are of the same version as the old ones.
- if the package with the same name was already installed, install-deps.sh won't reinstall it. see https://github.com/ceph/ceph/blob/b97022ce553d278c60524950ca3b16825dac0839/install-deps.sh#L136-L140
- i reinstalled the new package of ceph-libboost1.67-dev_1.67.0-7_amd64.deb on all ubuntu builders by reissuing "retest this please" command to jenkins, and ensuring all ubuntu builders are running the build job of https://github.com/ceph/ceph/pull/28245.
Updated by Nathan Cutler almost 5 years ago
- Assignee changed from Casey Bodley to Kefu Chai
Thanks for fixing the issue, Kefu! Can this be marked Resolved now?
Updated by Casey Bodley almost 5 years ago
- Assignee changed from Kefu Chai to Casey Bodley
Thanks Kefu, you are my hero.