Project

General

Profile

Bug #58382

centos 8 builds fail on main from cmake "Error executing java -version"

Added by Laura Flores about 1 year ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

https://shaman.ceph.com/builds/ceph/main/4d8388a3d8bbb62d5d3678728508a4487fc1e94c/default/328335/

-- Performing Test COMPILER_SUPPORTS_DIAGNOSTICS_COLOR - Success
CMake Error at /usr/share/cmake/Modules/FindJava.cmake:171 (message):
  Error executing java -version
Call Stack (most recent call first):
  src/java/CMakeLists.txt:1 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1677-g4d8388a3/rpm/el8/BUILD/ceph-18.0.0-1677-g4d8388a3/x86_64-redhat-linux-gnu/CMakeFiles/CMakeOutput.log".
See also "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1677-g4d8388a3/rpm/el8/BUILD/ceph-18.0.0-1677-g4d8388a3/x86_64-redhat-linux-gnu/CMakeFiles/CMakeError.log".
error: Bad exit status from /var/tmp/rpm-tmp.0ctaEn (%build)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.0ctaEn (%build)
+ rm -fr /tmp/install-deps.1646984
Build step 'Execute shell' marked build as failure

History

#1 Updated by Laura Flores about 1 year ago

This build from 12 days ago was the last one that was still okay.
https://shaman.ceph.com/builds/ceph/main/461ac2c30414f9a26b62c17532971b1eb2b1b18c/default/327611/

#2 Updated by Laura Flores about 1 year ago

This is what the output normally looks like, when cmake is successful:

-- Performing Test COMPILER_SUPPORTS_DIAGNOSTICS_COLOR - Success
-- Found Java: /usr/lib/jvm/java-1.8.0-openjdk/bin/java (found version "1.8.0.322") found components: Development 

#3 Updated by Laura Flores about 1 year ago

The difference is that the successful builds were installing java v1.8.0.322, while the newer builds have installed java v1.8.0.352.

#4 Updated by Laura Flores about 1 year ago

This seems to be the script that installs the dependencies based on distro, when a branch is pushed to shaman: https://github.com/ceph/ceph-build/blob/main/scripts/build_utils.sh

#5 Updated by Laura Flores about 1 year ago

-DJAVA_HOME is set to "/usr/lib/jvm/java-1.8.0-openjdk", which is where the installed java version should be.

According to the package site for ava v1.8.0.352 (https://centos.pkgs.org/8-stream/centos-appstream-x86_64/java-1.8.0-openjdk-1.8.0.352.b08-2.el8.x86_64.rpm.html), the filepath is "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el8.x86_64". Perhaps the more general filepath is somehow not getting created?

It's difficult to poke around and see since the sepia VPN is down right now, but that's a guess.

#6 Updated by Laura Flores about 1 year ago

I was able to reproduce this locally in a centos 8 container.

I simulated the situation manually by first installing ceph dependencies and running this cmake command (copy-pasted from the shaman build output):

[root@e217368e03f9 ceph]# cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR:PATH=/usr/libexec -DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/var -DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc -DCMAKE_INSTALL_MANDIR:PATH=/usr/share/man -DCMAKE_INSTALL_DOCDIR:PATH=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR:PATH=/usr/include -DSYSTEMD_SYSTEM_UNIT_DIR:PATH=/usr/lib/systemd/system -DWITH_MANPAGE:BOOL=ON -DWITH_PYTHON3:STRING=3.6 -DWITH_MGR_DASHBOARD_FRONTEND:BOOL=OFF -DJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk -DJAVA_LIB_INSTALL_DIR=/usr/lib/java -DWITH_CEPHFS_JAVA:BOOL=ON -DWITH_SELINUX:BOOL=ON -DWITH_LTTNG:BOOL=ON -DWITH_BABELTRACE:BOOL=ON -DALLOCATOR=tcmalloc -DWITH_SYSTEM_BOOST=OFF -DWITH_BOOST_VALGRIND=ON -DWITH_OCF:BOOL=ON -DWITH_LIBRADOSSTRIPER:BOOL=ON -DWITH_RADOSGW_AMQP_ENDPOINT:BOOL=ON -DWITH_RADOSGW_KAFKA_ENDPOINT:BOOL=ON -DWITH_RBD_RWL:BOOL=ON -DWITH_RBD_SSD_CACHE:BOOL=ON -DBOOST_J:STRING=48 -DWITH_FMT_HEADER_ONLY:BOOL=ON -DWITH_GRAFANA:BOOL=ON

During that process, cmake completes successfully, and the symlink `/usr/lib/jvm/java-1.8.0-openjdk` is created. I removed the symlink with `rm -rf rm -rf /usr/lib/jvm/java-1.8.0-openjdk`, and then re-ran the cmake command:

[root@e217368e03f9 ceph]# cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR:PATH=/usr/libexec -DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/var -DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc -DCMAKE_INSTALL_MANDIR:PATH=/usr/share/man -DCMAKE_INSTALL_DOCDIR:PATH=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR:PATH=/usr/include -DSYSTEMD_SYSTEM_UNIT_DIR:PATH=/usr/lib/systemd/system -DWITH_MANPAGE:BOOL=ON -DWITH_PYTHON3:STRING=3.6 -DWITH_MGR_DASHBOARD_FRONTEND:BOOL=OFF -DJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk -DJAVA_LIB_INSTALL_DIR=/usr/lib/java -DWITH_CEPHFS_JAVA:BOOL=ON -DWITH_SELINUX:BOOL=ON -DWITH_LTTNG:BOOL=ON -DWITH_BABELTRACE:BOOL=ON -DALLOCATOR=tcmalloc -DWITH_SYSTEM_BOOST=OFF -DWITH_BOOST_VALGRIND=ON -DWITH_OCF:BOOL=ON -DWITH_LIBRADOSSTRIPER:BOOL=ON -DWITH_RADOSGW_AMQP_ENDPOINT:BOOL=ON -DWITH_RADOSGW_KAFKA_ENDPOINT:BOOL=ON -DWITH_RBD_RWL:BOOL=ON -DWITH_RBD_SSD_CACHE:BOOL=ON -DBOOST_J:STRING=48 -DWITH_FMT_HEADER_ONLY:BOOL=ON -DWITH_GRAFANA:BOOL=ON
CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.

 '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--cache-file=../config.cache' '--disable-static' '--enable-symbol-hiding' '--enable-ipv6' '--enable-threaded-resolver' '--without-libmetalink' '--with-gssapi' '--with-nghttp2' '--with-ssl' '--with-ca-bundle=/etc/pki/tls/certs/ca-bundle.crt' '--enable-ldap' '--enable-ldaps' '--enable-manual' '--with-brotli' '--with-libidn2' '--with-libpsl' '--with-libssh' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
-- libcurl is linked with openssl: explicitly setting locks
-- ssl soname: libssl.so.1.1
-- crypto soname: libcrypto.so.1.1
-- BUILDING Boost Libraries at j 48
-- boost will be downloaded...
-- Found nasm: best -- capable of assembling AVX512
CMake Error at /usr/share/cmake/Modules/FindJava.cmake:171 (message):
  Error executing java -version
Call Stack (most recent call first):
  src/java/CMakeLists.txt:1 (find_package)

-- Configuring incomplete, errors occurred!
See also "/root/ceph/CMakeFiles/CMakeOutput.log".
See also "/root/ceph/CMakeFiles/CMakeError.log".

The error is reproduced. Somehow, the `/usr/lib/jvm/java-1.8.0-openjdk` symlink isn't present, and cmake can't find java.

#7 Updated by Laura Flores about 1 year ago

I reset the symlink in my local environment with `ln -s /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64 /usr/lib/jvm/java-1.8.0-openjdk`, and cmake made it past that error.

#8 Updated by Laura Flores about 1 year ago

The symlink is created right after package install:

[root@49453413486a ceph]# ls /usr/lib/jvm
java  java-1.8.0  java-1.8.0-openjdk  java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64  java-openjdk

#9 Updated by Laura Flores about 1 year ago

This line in the build script used to install dependenies (https://github.com/ceph/ceph-build/blob/main/scripts/build_utils.sh) shows that the dependencies come from the "ceph.spec.in" file from the cloned ceph repo. This could provide some clues on what dependencies are being installed.

sed -e 's/@//g' < ceph.spec.in > $DIR/ceph.spec

#10 Updated by Laura Flores about 1 year ago

Got a bit further to understanding what's wrong.

With this change to ceph.spec.in:

--- a/ceph.spec.in
+++ b/ceph.spec.in
@@ -196,7 +196,7 @@ Requires:       ceph-mgr = %{_epoch_prefix}%{version}-%{release}
 Requires:       ceph-mon = %{_epoch_prefix}%{version}-%{release}
 Requires(post):        binutils
 %if 0%{with cephfs_java}
 BuildRequires: java-devel
 BuildRequires: jpackage-utils
 BuildRequires: sharutils
 %endif
@@ -1353,8 +1353,6 @@ cmake .. \
     -DWITH_TESTS:BOOL=OFF \
 %endif
 %if 0%{with cephfs_java}
-    -DJAVA_HOME=%{java_home} \
-    -DJAVA_LIB_INSTALL_DIR=%{_jnidir} \
     -DWITH_CEPHFS_JAVA:BOOL=ON \
 %endif
 %if 0%{with selinux}

I got cmake to detect a java version, albeit not the correct one:

-- Performing Test COMPILER_SUPPORTS_DIAGNOSTICS_COLOR - Success
-- Found Java: /usr/bin/java (found version "1.8.0.322") found components: Development 
-- Found JNI: /usr/lib/jvm/java/jre/lib/aarch64/libjawt.so  

Note that 1.8.0.322 is the older Java version we were installing, and now we're installing 1.8.0.352.

Later on, the build failed when trying to install Boost, so the solution above is not correct:

[  7%] Completed 'Boost'
[  7%] Built target Boost
make: *** [Makefile:146: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.kN0QLI (%build)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.kN0QLI (%build)
+ rm -fr /tmp/install-deps.2467935
Build step 'Execute shell' marked build as failure
New run name is '#67159 origin/wip-lflores-testing, da31a0767446ad67dbe0124bfa302d675b26c7ac, jammy focal centos8 centos9 windows, default'
[PostBuildScript] - [INFO] Executing post build scripts.
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties file path '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/build_info'
[EnvInject] - Variables injected successfully.
[gigantic] $ /bin/bash /tmp/jenkins3374889353428547710.sh
+ cd /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic
+ sudo rm -rf ceph-container
++ branch_slash_filter origin/wip-lflores-testing
++ RAW_BRANCH=origin/wip-lflores-testing
+++ grep -o /
+++ wc -l
++ branch_slashes=1
+++ echo origin/wip-lflores-testing
+++ rev
+++ cut -d / -f 1
+++ rev
++ FILTERED_BRANCH=wip-lflores-testing
++ '[' 1 -gt 1 ']'
++ echo wip-lflores-testing
+ BRANCH=wip-lflores-testing
+ failed_build_status ceph centos 8 arm64
+ project=ceph
+ state=failed
+ distro=centos
+ distro_version=8
+ distro_arch=arm64
+ submit_build_status POST failed ceph centos 8 arm64
+ http_method=POST
+ state=failed
+ project=ceph
+ distro=centos
+ distro_version=8
+ distro_arch=arm64
+ cat
+ cat
+ SHAMAN_URL=https://shaman.ceph.com/api/builds/ceph/
+ curl -X POST -H Content-Type:application/json --data @/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/build_status.json -u admin:**** https://shaman.ceph.com/api/builds/ceph/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   814  100     2  100   812     10   4342 --:--:-- --:--:-- --:--:--  4352
{}[Checks API] No suitable checks publisher found.
Finished: FAILURE

But it indicates that we have multiple java versions installed on centos 8 nodes.

See the full log of the output I pasted here: https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/67159//consoleFull

#11 Updated by Laura Flores about 1 year ago

Latest developments:

When cmake tries to execute `java -version` with the latest java version, we get this error:

[lflores@braggi04 ~]$ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el8.x86_64/bin/java -version
Error: could not find libjava.so
Error: Could not find Java SE Runtime Environment.

The missing library, "libjava.so" (https://pkgs.org/download/libjava.so) appears to come from a package called "java-1.8.0-openjdk-headless" (https://centos.pkgs.org/8-stream/centos-appstream-x86_64/java-1.8.0-openjdk-headless-1.8.0.352.b08-2.el8.x86_64.rpm.html).

On one of the affected nodes, we can see that "java-1.8.0-openjdk-headless" is the only package that is still version 322 rather than the newer 352 version.

[lflores@braggi04 ~]$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-devel-1.8.0.352.b08-2.el8.x86_64
java-1.8.0-openjdk-headless-1.8.0.322.b06-11.el8.x86_64
java-1.8.0-openjdk-1.8.0.352.b08-2.el8.x86_64

A possible solution would be to install the newer "java-1.8.0-openjdk-headless" version manually, with

sudo dnf install java-1.8.0-openjdk-headless

But, we believe this package should have been installed alongside "java-1.8.0-openjdk-devel" in the first place. Why the new version of "java-1.8.0-openjdk-headless" was not installed along with "java-1.8.0-openjdk-devel" is still a mystery.

#12 Updated by Samuel Just about 1 year ago

Looks like ceph.spec.in specifies java-devel jpackage-utils and sharutils. java-devel is presumably pulling java-1.8.0-openjdk-devel and java-1.8.0-openjdk. Perhaps there's a packaging error (version requirement?) where either java-1.8.0-openjdk or java-devel should be pulling java-1.8.0-openjdk-headless but it isn't? What if we hard-specify openjdk-headless in ceph.spec.in?

#13 Updated by Samuel Just about 1 year ago

Are those machines running centos 8 or centos stream 8? Presumably the latter?

#14 Updated by Laura Flores about 1 year ago

Samuel Just wrote:

Are those machines running centos 8 or centos stream 8? Presumably the latter?

centos stream 8

#15 Updated by Samuel Just about 1 year ago

https://github.com/ceph/ceph/pull/49711 disables cephfs_java for the time being. The only user is the cephfs jni bindings, so it shouldn't really pose a problem.

#16 Updated by Dan Mick about 1 year ago

I don't understand exactly why it was failing, but I managed to update all the Centos8 nodes to have v352 packages by 1) dnf remove java-1.8.0-openjdk-headless and 2) dnf install java-1.8.0-openjdk. java -version works on all the nodes now, and I did a build (of a branch with a meaningless commit, skipping container build) at https://jenkins.ceph.com/job/ceph-dev-new-build/67239/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/consoleFull that has finished successfully.

#17 Updated by Casey Bodley about 1 year ago

  • Status changed from New to Resolved

the centos 8 x86_64 builds are succeeding again, so i'm changing status to resolved. thanks to everybody that helped with this!

#18 Updated by Laura Flores 10 months ago

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/70444//consoleFull

[lflores@braggi06 ~]$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-devel-1.8.0.362.b08-3.el8.x86_64
java-1.8.0-openjdk-headless-1.8.0.352.b08-2.el8.x86_64
java-1.8.0-openjdk-1.8.0.362.b08-3.el8.x86_64

Also available in: Atom PDF