Bug #16876
java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/jni/libcephfs_jni.so
0%
Description
I had a failed fs testsuite run, and a couple of the jobs failed with what looks like the error below:
2016-07-30T18:05:59.536 INFO:tasks.workunit.client.0.mira119.stdout:.Loading libcephfs-jni from default path: /usr/lib/jni:/usr/lib64 2016-07-30T18:05:59.537 INFO:tasks.workunit.client.0.mira119.stdout:Loading libcephfs-jni: /usr/lib64/libcephfs_jni.so 2016-07-30T18:05:59.537 INFO:tasks.workunit.client.0.mira119.stdout:Loading libcephfs-jni: /usr/lib/jni/libcephfs_jni.so 2016-07-30T18:05:59.537 INFO:tasks.workunit.client.0.mira119.stdout:Loading libcephfs-jni: Failure! [...] ) 2016-07-30T18:05:59.550 INFO:tasks.workunit.client.0.mira119.stdout:Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/jni/libcephfs_jni.so 2016-07-30T18:05:59.550 INFO:tasks.workunit.client.0.mira119.stdout: at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1827) 2016-07-30T18:05:59.550 INFO:tasks.workunit.client.0.mira119.stdout: at java.lang.Runtime.load0(Runtime.java:809) 2016-07-30T18:05:59.550 INFO:tasks.workunit.client.0.mira119.stdout: at java.lang.System.load(System.java:1086) 2016-07-30T18:05:59.550 INFO:tasks.workunit.client.0.mira119.stdout: at com.ceph.fs.CephNativeLoader.loadLibrary(CephNativeLoader.java:76) 2016-07-30T18:05:59.551 INFO:tasks.workunit.client.0.mira119.stdout: at com.ceph.fs.CephMount.loadLibrary(CephMount.java:102) 2016-07-30T18:05:59.551 INFO:tasks.workunit.client.0.mira119.stdout: at com.ceph.fs.CephMount.<clinit>(CephMount.java:98) 2016-07-30T18:05:59.551 INFO:tasks.workunit.client.0.mira119.stdout: at com.ceph.fs.CephDoubleMountTest.test_double_mount(CephDoubleMountTest.java:33) 2016-07-30T18:05:59.551 INFO:tasks.workunit.client.0.mira119.stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2016-07-30T18:05:59.552 INFO:tasks.workunit.client.0.mira119.stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2016-07-30T18:05:59.552 INFO:tasks.workunit.client.0.mira119.stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2016-07-30T18:05:59.552 INFO:tasks.workunit.client.0.mira119.stdout: at java.lang.reflect.Method.invoke(Method.java:498) 2016-07-30T18:05:59.552 INFO:tasks.workunit.client.0.mira119.stdout: at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) 2016-07-30T18:05:59.552 INFO:tasks.workunit.client.0.mira119.stdout: at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 2016-07-30T18:05:59.553 INFO:tasks.workunit.client.0.mira119.stdout: at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) 2016-07-30T18:05:59.553 INFO:tasks.workunit.client.0.mira119.stdout: at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) 2016-07-30T18:05:59.553 INFO:tasks.workunit.client.0.mira119.stdout: at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19) 2016-07-30T18:05:59.553 INFO:tasks.workunit.client.0.mira119.stdout: ... 31 more
The two tests that failed with this are here:
http://pulpito.ceph.com/jlayton-2016-07-29_18:51:42-fs-wip-jlayton-nlink---basic-mira/341523
http://pulpito.ceph.com/jlayton-2016-07-29_18:51:42-fs-wip-jlayton-nlink---basic-mira/341567
The branch is fairly recent master branch (late last week) with a few relatively small userland c++ client bugfixes.
It looks like the problem is that the libcephfs_jni.so symlink is missing:
[jlayton@mira119 ~]$ ls -l /usr/lib64/libcephfs_jni* lrwxrwxrwx. 1 root root 22 Jul 31 20:27 /usr/lib64/libcephfs_jni.so.1 -> libcephfs_jni.so.1.0.0 -rwxr-xr-x. 1 root root 111552 Jul 25 14:53 /usr/lib64/libcephfs_jni.so.1.0.0
Seems like this loader ought to be opening libcephfs_jni.so.1. You may not have the right symbols if the library version moves to .2 (or later).
Related issues
History
#1 Updated by Jeff Layton over 7 years ago
Of course, it may be that I reached the box too late and the filesystem had been changed. I'm not sure how to tell. Either way though, I think the point of using the .so.X symlink to do the loading instead of the .so one is valid. In most fedora/rhel packages at least, the .so symlink is part of the -devel package. So you really don't want to rely on that for anything at runtime (assuming the jni loader isn't doing that for specific reasons).
#2 Updated by John Spray over 7 years ago
- Duplicates Bug #16640: libcephfs: Java bindings failing to load on CentOS added
#3 Updated by John Spray over 7 years ago
- Status changed from New to Duplicate
This should be fixed in master now (there was a backed-out change for 16640 in teuthology, then finally the fix was https://github.com/ceph/ceph-qa-suite/pull/1084)