#54 rpath and runpath set to $ORIGIN are not handled correctly

[Vincent Danjean]

Hi,

I'm trying to run OpenCL programs under bumblebee. When using the Intel OpenCL implementation, I found a bug that I tracked into libdlfaker.so.

Here is the run of a little OpenCL program that only print locally installed OpenCL implementation:
$ ./ocl_test
Found 3 platforms!
Advanced Micro Devices, Inc.
Intel(R) Corporation
ocl-icd ICD test

When run with vgl LD_PRELOAD libraries (which is what occurs in bumblebee), here are the results (I enabled verbose/trace messages):
$ VGL_VERBOSE=1 VGL_TRACE=1 LD_PRELOAD=/tmp/inst/opt/VirtualGL/lib/libdlfaker.so:/tmp/inst/opt/VirtualGL/lib/librrfaker.so ./ocl_test
[VGL] dlopen (filename=libamdocl64.so flag=1 retval=0x01e22320)
[VGL] dlopen (filename=/usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel/libintelocl.so flag=1[VGL] dlopen (filename=libtbbmalloc.so.2 flag=2 retval=0x01e55ae0)
[VGL] dlopen (filename=NULL flag=1 retval=0x7f4068dea1c8)
[VGL] dlopen (filename=libirml.so.1 flag=1 retval=0x00000000)
retval=0x01e542b0)
[VGL] dlopen (filename=/home/vdanjean/debian/mainteneur/ocl-icd/upstream/ocl-icd/build/.libs/libdummycl.so flag=1 retval=0x01e580a0)
[VGL] dlopen (filename=libaticaldd.so flag=2 retval=0x00000000)
[VGL] dlopen (filename=libclang_compiler.so flag=1 retval=0x00000000)
[VGL] dlopen (filename=libcpu_device.so flag=1 retval=0x00000000)
Segmentation fault (core dumped)

Note that /tmp/inst/opt/VirtualGL/lib/libdlfaker.so and /tmp/inst/opt/VirtualGL/lib/librrfaker.so come from a fresh local recompilation of https://virtualgl.svn.sourceforge.net/svnroot/virtualgl/vgl/trunk revision 5405. So the bug is really in VirtualGL, not in bumblebee.

You can see that the last three dlopen return NULL.

The bug can be workaround by setting correctly LD_LIBRARY_PATH:
$ LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel VGL_VERBOSE=1 VGL_TRACE=1 LD_PRELOAD=/tmp/inst/opt/VirtualGL/lib/libdlfaker.so:/tmp/inst/opt/VirtualGL/lib/librrfaker.so ./ocl_test
[VGL] dlopen (filename=libamdocl64.so flag=1 retval=0x02354320)
[VGL] dlopen (filename=/usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel/libintelocl.so flag=1[VGL] dlopen (filename=libtbbmalloc.so.2 flag=2 retval=0x02387a40)
[VGL] dlopen (filename=NULL flag=1 retval=0x7feef89601c8)
[VGL] dlopen (filename=libirml.so.1 flag=1 retval=0x00000000)
retval=0x023862b0)
[VGL] dlopen (filename=/home/vdanjean/debian/mainteneur/ocl-icd/upstream/ocl-icd/build/.libs/libdummycl.so flag=1 retval=0x0238a000)
[VGL] dlopen (filename=libaticaldd.so flag=2 retval=0x00000000)
[VGL] dlopen (filename=libclang_compiler.so flag=1 retval=0x02395220)
[VGL] dlopen (filename=libcpu_device.so flag=1 retval=0x0239e2b0)
Found 3 platforms!
Advanced Micro Devices, Inc.
Intel(R) Corporation
ocl-icd ICD test

The root cause is that these dlopen are done in the libintelocl.so library that has these features:
$ readelf -a /usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel/libintelocl.so | grep path
0x000000000000000f (RPATH) Library rpath: [$ORIGIN]
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN]
When these dlopen are diverted by libdlfaker.so, they are re-run from libdlfaker.so itself, so the directory where libintelocl.so is put is not used anymore by __dlopen. So the calls fail.

I do not see an easy way to solve this bug, but you must be aware that it breaks any OpenCL programs run through VirtualGL (or bumblebee) if the Intel OpenCL implementation is installed. You can look at my initial messages on the Intel forum when I was thinking that the bug were in Intel ICD:
http://software.intel.com/en-us/comment/1656004

Regards,
Vincent

[DRC]

I assume it works with 'vglrun -nodl'. dlfaker is enabled by default because there were enough 3D applications that required it that I was having to field one too many queries regarding why an app wouldn't work, when all the user had to do was run it with vglrun -dl. At some point, the number of apps requiring it far exceeded those that required it to be disabled, but this sort of problem is exactly the sort of problem that -nodl is intended for. I don't know of any straightforward way to address it in the faker itself. Closing as WNF, but please re-open if you need to add more data or if, for some reason, -nodl doesn't work.

[DRC]

If you can verify that -nodl works around the problem, I will add this as an "application recipe" in the docs.

[DRC]

Actually, I can't reproduce this. I installed the Intel OpenCL SDK 2012 (http://software.intel.com/en-us/vcsource/tools/opencl-sdk) on CentOS 6 and compiled this program:

http://svn.clifford.at/tools/trunk/examples/cldemo.c

using

c++ -O3 -o cldemo cldemo.c -lOpenCL

Runs fine with vglrun. I also tried moving the faker libs to a temp directory (/tmp/vgl) and explicitly specifying LD_PRELOAD as you did above, rather than using vglrun. Still runs fine. Apparently something is different about our systems.

I verified that the Red Hat version of libintelocl.so also has RPATH and RUNPATH set to $ORIGIN. Based on the location of libintelocl on your system, I assume you're using Ubuntu, although you didn't say as much.

[DRC]

I was finally able to reproduce this. See https://sourceforge.net/p/virtualgl/mailman/message/35185340/. I'm open to any advice as to how to fix it, but I personally can't think of anything other than the workaround you proposed, which can be easily implemented by running the OpenCL application with:

vglrun -ld /opt/intel/opencl/lib64 {application}

I've documented this in the app. recipes section of the User's Guide.

Unless the application is using OpenGL, or a GPU-based implementation of OpenCL, then using vglrun shouldn't be necessary. If the application is using OpenGL but is not loading libGL with dlopen() (that is, if the application directly links with libGL), then you can also use vglrun -nodl as a workaround.