Ceph » rgw

Maybe this is the root cause? #39981

All of the console logs are from a single PR: https://github.com/ceph/ceph/pull/19371

And "make check" just passed in that PR following merge of https://github.com/ceph/ceph/pull/28181 which is. . . a positive sign.

https://github.com/ceph/ceph/pull/28058

Still happening: https://github.com/ceph/ceph/pull/28211

So I'm removing the related-to link to #39981.

https://jenkins.ceph.com/job/ceph-pull-requests/556/console

https://jenkins.ceph.com/job/ceph-pull-requests/555/console

https://jenkins.ceph.com/job/ceph-pull-requests/573/console

I set up a fresh bionic vm and was able to build master successfully. The ubuntu flavors of master and nautilus are still succeeding in https://shaman.ceph.com/builds/ceph/. It looks like the difference here is the use of WITH_SYSTEM_BOOST and the ceph-libboost-* packages.

From the logs of failures, I do see that cmake is able to find the boost context component:

-- Boost version: 1.67.0
-- Found the following Boost libraries:
--   atomic
--   chrono
--   thread
--   system
--   regex
--   random
--   program_options
--   date_time
--   iostreams
--   python27
--   context
--   coroutine
--   filesystem
--   timer

But the failure implies that libboost_context.so isn't present under /opt/ceph/:

/usr/bin/ld: warning: libboost_context.so.1.67.0, needed by //opt/ceph/lib/x86_64-linux-gnu/libboost_coroutine.so.1.67.0, not found (try using -rpath or -rpath-link)

The logs of failures seem to be using xenial:

++ VERSION='16.04.4 LTS (Xenial Xerus)'

cmake -DWITH_SYSTEM_BOOST=ON -DBOOST_ROOT=/opt/ceph ..

Maybe just a shot in the dark, but can't the problem be in the libboost-* libs being installed? The "undefined reference" error occurs in libboost_coroutine.so.1.67.0, which is installed from some https://chacra.ceph.com/repos/... URL. Are these libs sane, or could they be already broken?

td;dr ceph-libboost1.67-dev_1.67.0-7_amd64.deb was not sane.

long story,

1. before https://github.com/tchaikov/ceph-boost/commit/ec205d3f54258c0d7f9d22ae279b1a20523e7629 was merged, we could consistently reproduce this issue.
2. I tried to fix it by rebuilding the ceph-libboost* packages and uploading them to chacra. but i failed to reinstall them to all xenial builders.
3. and re-running install-deps.sh cannot help because:
   • i was using the same changelog file when rebuilding the ceph-libboost* packages. as i thought it would be simpler to fix the builders than updating install-deps.sh, if the ceph-libboost* packages went wrong. so the new ceph-libboost* packages are of the same version as the old ones.
   • if the package with the same name was already installed, install-deps.sh won't reinstall it. see https://github.com/ceph/ceph/blob/b97022ce553d278c60524950ca3b16825dac0839/install-deps.sh#L136-L140
4. i reinstalled the new package of ceph-libboost1.67-dev_1.67.0-7_amd64.deb on all ubuntu builders by reissuing "retest this please" command to jenkins, and ensuring all ubuntu builders are running the build job of https://github.com/ceph/ceph/pull/28245.