#202 Matlab R2011b mex files crash with IT++

[Anonymous]

We found that (under Linux) Matlab mex files using IT++ crash since Matlab R2011b. After we have filed a support request at Mathworks it turned out that (see response attach at the end of this message) the reason for the crash is that since R2011b, Matlab's MKL (and hence the BLAS/LAPACK routines) expects 64-bit integers. When IT++ code is executed in a mex file it dynamically links against Matlab's MKL and hence crashes whenever BLAS/LAPACK routines are called since IT++ uses 32-bit integers.

I have created a patch (which is attached) to make sure that all BLAS and LAPACK routines are called with 64-bit integers. The patch applies cleanly against IT++ 4.2. All changes outside of itpp/base just fix some unused variable compile warnings. "make check" may fail at the tests which use BLAS/LAPACK if the local version of BLAS/LAPACK expects 32-bit integers.

We have now solved the problem for us by installing 2 versions of IT++, one using 32-bit integers (for stand-alone programs) and one using 64-bit integers (for mex programs). It might be possible to make a single version which can handle different MKL/BLAS/LAPACK versions (regarding integer width). To this end, we think that one would have to tackle the following two questions:

• Is it possible to detect at run-time if a program is executed stand-alone or as a mex file from within Matlab?

• Can a test be added to the configure script which checks if the local version of MKL/BLAS/LAPACK expects 64-bit or 32-bit integers? (This would be just a matter of detecting a SIGSEGV if the wrong integer width has been used.)

Best regards,
Andreas Winkelbauer

Response to our support request:

Because libitpp is dynamically linked against BLAS and LAPACK, at runtime of the MEX-file, automatically the MKL shipped with MATLAB will be used to provide BLAS and LAPACK functionality. There is a mismatch however between the integer datatypes used in the MKL shipped with MATLAB 7.13 (R2011b) (64-bit integers) and the integer datatypes used in libitpp (32-bit integers). This mismatch leads to a crash.

Now there are various possible solutions to this issue:

1. libitpp appears to be compilable without BLAS and LAPACK (--without-blas and --without-lapack options for the ./configure script). With these disabled the MEX-file works fine. This may have a negative impact on the performance of libitpp though.

2. You can force MATLAB to use your system's BLAS and LAPACK libraries (which work with 32-bit integers) instead of the MKL. To accomplish this create two environment variables BLAS_VERSION and LAPACK_VERSION and point them to your system's BLAS and LAPACK libraries respectively. So this could for example look like:

export BLAS_VERSION=/usr/local/blas.so
export LAPACK_VERSION=/usr/local/lapack.so
/usr/local/MATLAB/R2011b/bin/matlab

This may have a negative impact on MATLAB's performance though.

1. The source of libitpp could be updated to use 64-bit integers instead of 32-bit integers whenever it calls BLAS/LAPACK functionality. This will require some inside knowledge of libitpp though and probably can be a lengthy task.

[Anonymous]

patch for using 64-bit integers for BLAS/LAPACK routines

[Bogdan Cristea]

Could you provide more details about how you have compiled your mex files on Linux ? I am especially interested in how you solved libstdc++.so.6 library dependencies. Here is the full error message:

Invalid MEX-file '/home/bogdan/C++/IT++/mex/test.mexa64': /usr/local/MATLAB/R2012a/bin/glnxa64/libstdc++.so.6:
version `GLIBCXX_3.4.15' not found (required by /usr/local/lib64/libitpp.so.8)

[Anonymous]

Hi!

I compile the mex files using the following command:

mex -v -largeArrayDims CXXFLAGS="\$CXXFLAGS -O3 -pipe -pedantic -Wall" CXXLIBS="\$CXXLIBS /opt/itpp/lib64/libitpp.so -litpp" my_source.cpp

Here, /opt/itpp is the path where the modified ITPP is installed. In order to make matlab use this version of the library, I start it with

LD_LIBRARY_PATH=/opt/itpp/lib64:$LD_LIBRARY_PATH matlab

The problems with executing the mex files is due to the fact that matlab ships its own versions of libstdc++, libgfortran and others in /path/to/matlab/sys/os/glnxa64/. Hence, you have to force matlab to use the versions of these libraries that ship with your system (i.e. the ones that have been used to compile ITPP). One way to achieve this is to simply rename the relevant libraries in /path/to/matlab/sys/os/glnxa64/.

Best,
Andreas

[Andreas Winkelbauer]

This problem still exists in IT++ 4.3.1.

[Andreas Winkelbauer]

On a related note: All currently available versions of Matlab >= 8.x under Linux are basically unusable due to the following bug:

http://www.mathworks.de/support/bugreports/961964
https://stackoverflow.com/questions/19268293/matlab-error-cannot-open-with-static-tls

This is particularly true when using larger IT++ mex files. When such a mex file is executed, the following error will likely occur:

dlopen: cannot load any more object with static TLS

[Bogdan Cristea]

Hi
The provided patch has not been integrated because we don't have the capability to check IT++ after applying the patch. Are you available to run some tests on your side with MATLAB after patch application in the master branch ?
thanks
Bogdan

[Andreas Winkelbauer]

Sure, I can do that.

However, note that applying the patch as is will break standalone IT++ applications on systems where the BLAS and LAPACK routines expect 32-bit integers.

[Bogdan Cristea]

Please update your patch to allow both 32 and 64 systems.

[Andreas Winkelbauer]

Well, this is highly nontrivial as far as I can tell. The problem is not with 32-bit or 64-bit operating systems, but rather with the integer width used in the BLAS and LAPACK routines. The libraries that come with most Linux distributions use 32-bit integers whereas newer MKL libraries (e.g., in MATLAB) use 64-bit integers.

A patch which would work with both integer widths would have to introduce some logic allowing IT++ to detect at runtime which integer width is expected by the BLAS and LAPACK routines. Currently I do not have an idea how this could be implemented.

In case you know how to do this or in case you have an alternative solution, please let me know.

[Bogdan Cristea]

I am thinking at a simpler solution to let the user decide which type of library she wants to compile and provide in the code some ifdefs. By default the 32 bits version should be compiled. You need to put an option in the cmake file in order to allow the user to select which library will be generated. I can help you with that part.

[Andreas Winkelbauer]

OK, that sounds feasible. I'll get back to you.

[Andreas Winkelbauer]

Hi!

I revisited this issue and now I have a set of three patches which you might want to consider for a merge.

As discussed previously, the idea is to enable the user to compile IT++ with different integer widths for the BLAS and LAPACK calls. The user can select between 32 bit integers (default; unchanged behavior compared to current IT++) and 64 bit integers (e.g., for using IT++ with the BLAS and LAPACK routines provided by MATLAB).

To achieve this, I have made the following changes (each change corresponds to one patch):

1) I have added the cmake options -DBLAS_INT and -DLAPACK_INT which can be set to 32 or 64. If the options are not specified, 32 is assumed which is identical to the current version of IT++. The build system then adds -DBLAS_INT32 or -DBLAS_INT64 (similarly, -DLAPACK_INT32 or -DLAPACK_INT64) to the g++ compiler calls.

I kept the integer width for BLAS and LAPACK separate to allow for mixed integer widths which gives a more fine-grained control. However, usually -DBLAS_INT and -DLAPACK_INT will both either be equal to 32 or to 64.

2) I defined the data types int_blas_t and int_lapack_t which, depending on the configuration, correspond to 32 bit integers or 64 bit integers. Introducing these data types allows me to perform the remaining code changes without any additional conditional compilation. Otherwise, the code would have been polluted with #if ... #else ... #endif statements which facilitates neither readability nor maintainability.

3) The largest part of the patch concerns the code changes around the BLAS and LAPACK calls. The nice thing here is that this code unifies the original code and my (old) previous patch. Furthermore, this part of the code may remain unchanged even if additional other integer widths are added in the future (only the first two small patches would need to be changed/adapted).

The patches apply cleanly against the latest version in the public git repo, i.e., on top of commit 6a075f307e729c85522dbb77d91b606b7031c7e1.

I have thoroughly tested my patches. Furthermore, all tests pass in either configuration.

I will post the patches and more details below.

[Andreas Winkelbauer]

Add cmake options -DBLAS_INT and -DLAPACK_INT. Both options can be set to 32 (which is the default) or 64. This allows us to compile IT++ for BLAS and LAPACK libraries which expect integer widths other than 32 bits.

I have made these changes in the top-level CMakeLists.txt file. If you prefer, I could modify the patch to introduce the changes in the CMakeLists.txt file of the itpp subdirectory.

[Andreas Winkelbauer]

Define the data types int_blas_t and int_lapack_t. This enables us to select (at compile-time) the integer width that is expected by the BLAS and LAPACK library, respectively.

This patch contains all the conditional compilation (#if/#else/#endif) statements that are required. In case additional integer widths shall be added, only this and the previous patch need to be changed (the next, large patch may remain unchanged).

[Andreas Winkelbauer]

Use the data types int_blas_t and int_lapack_t in all calls to the BLAS and LAPACK libraries. This allows us to compile IT++ for BLAS and LAPACK libraries which expect integer widths other than 32 bits.

This patch unifies the current IT++ code and my old patch.

[Bogdan Cristea]

Thanks Andreas for your patch, let me look at your changes first.

[Andreas Winkelbauer]

I have tested the 32 bit configuration as follows:

$ mkdir build && cd build
$ cmake -DGTEST_DIR=/path/to/gtest -DHTML_DOCS=off -DCMAKE_INSTALL_PREFIX=/usr ..
$ make
$ ./gtests/itpp_gtests

--> test result <--
[----------] Global test environment tear-down
[==========] 78 tests from 65 test cases ran. (40110 ms total)
[ PASSED ] 78 tests.
--> test result <--

$ sudo make install

Furthermore, standalone applications (see the example in the attachment) can be compiled and executed as follows:

$ g++ -c -ansi -D_GNU_SOURCE -O3 -pedantic -Wall itpp_test.cpp
$ g++ -O -o itpp_test itpp_test.o -lm -litpp
$ ./itpp_test

A MATLAB mex file which uses IT++ can be executed by telling MATLAB to use the system's BLAS and LAPACK libraries (which expect 32 bit integers). To this end, MATLAB has to be started as follows:

$ BLAS_VERSION=/usr/lib/libblas.so LAPACK_VERSION=/usr/lib/liblapack.so /path/to/bin/matlab

Tests under Windows and Mac OS are most welcome, as I only use Linux.

[Andreas Winkelbauer]

To test the 64 bit configuration, I have configured IT++ to use the BLAS and LAPACK libraries provided by MATLAB (which expect 64 bit integers; otherwise the tests would use the system's BLAS and LAPACK and would therefore fail).

$ mkdir build && cd build
$ cmake -DGTEST_DIR=/path/to/gtest -DBLAS_INT=64 -DLAPACK_INT=64 -DHTML_DOCS=off -DCMAKE_INSTALL_PREFIX=/usr -DBLAS_LIBRARIES="/path/to/matlab/bin/glnxa64/libmkl.so" -DLAPACK_LIBRARIES="/path/to/matlab/sys/os/glnxa64/libiomp5.so" -DFFT_LIBRARIES="/usr/lib/x86_64-linux-gnu/libfftw3.so" ..
$ make
$ ./gtests/itpp_gtests

--> test result <--
[----------] Global test environment tear-down
[==========] 78 tests from 65 test cases ran. (39648 ms total)
[ PASSED ] 78 tests.
--> test result <--

$ sudo make install

Furthermore, standalone applications can be compiled and executed as follows:

$ g++ -c itpp_test.cpp
$ g++ -O -o itpp_test itpp_test.o -lm -litpp -lfftw3 -Wl,-rpath,/path/to/matlab/bin/glnxa64:/path/to/matlab/sys/os/glnxa64/:
$ ./itpp_test

To execute MATLAB mex files which use IT++, no changes are necessary in this case (just start MATLAB normally).

In either case, the mex file can be compiled in MATLAB as follows:

mex CXXLIBS="\$CXXLIBS -litpp" source_file.cpp

[Bogdan Cristea]

Hi

I have committed your changes into master branch of git repository. Tests pass on OS X Yosemite. We still need to make sure that on Windows everything works as expected.

thanks
Bogdan

[Andreas Winkelbauer]

Thanks!

I have another patch with trivial changes (mostly unused variables). Should I open a separate bug for that?

Also: It seems that itpp/base/algebra/lapack.h is executable. Maybe you can remove the execute permissions on this file.

[Bogdan Cristea]

Hi

Please open another bug report and attach your patch. Actually there are a lot of files that should be removed (everything related to the old build system).

regards
Bogdan