#20 Dynamic Library not linked

[Jaime Souza]

Hi all,

I have been testing DAETOOLS on Windows and Linux. Now I am using MAC OS. After installing with system's Python3.6 (not conda or anaconda) and vitualenv, I get this problem when I execute the whats_the_time.py tutorial:

(.venv) ~/PycharmProjects/study-daetools(master ✗) python whats_the_time.py                                           
Traceback (most recent call last):
  File "whats_the_time.py", line 72, in <module>
    from daetools.pyDAE import *
  File "/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/daetools-1.8.0-py3.6-macosx-10.6-intel.egg/daetools/pyDAE/__init__.py", line 20, in <module>
    import pyCore
ImportError: dlopen(/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/daetools-1.8.0-py3.6-macosx-10.6-intel.egg/daetools/pyDAE/Darwin_x86_64_py36/pyCore.so, 2): Library not loaded: @rpath/libpython3.6m.dylib
Referenced from: /Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/daetools-1.8.0-py3.6-macosx-10.6-intel.egg/daetools/pyDAE/Darwin_x86_64_py36/pyCore.so
  Reason: image not found

This missing dylib is in "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib".

Thanks,

Jaime

[Anonymous]

Hi Jaime,
When one runs the python script with extension modules, in a normal case first the Python binary loads the libpython, then the extension modules load the libpython, which is already loaded and everything is fine. If the extension module can't find it, that means the libpython used by the python binary you used to run whats_the_time.py is not the same as libpython used by pyCore. That can be caused by different reasons. Perhaps python command points to a different version. Can you first check the sys.path and let me know? That will tell us what is the python you used. Once we know the sys path I can try to see what is the problem. m in libpython3.6m defines the ABI. If the python version you used match the version in pyCore, perhaps the ABI does not match.

Cheers
Dragan

[Jaime Souza]

Thanks, Dragan,

Here it is:

>>> import sys
>>> print(sys.path)
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages', '/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/setuptools-39.0.1-py3.6.egg', '/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/pip-9.0.3-py3.6.egg', '/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/pygraphviz-1.4rc1-py3.6-macosx-10.6-intel.egg', '/Users/jaimenms/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/daetools-1.8.0-py3.6-macosx-10.6-intel.egg']

The same result if I include this print in the whats_the_time.py file.

And this is the only Python3 version I have:

(.venv) ~/PycharmProjects/study-daetools(master ✗) python --version
Python 3.6.5

Regards

Jaime

[ciroki]

Hi Jaime,

Something is not standard in your installation. The path to libpython you sent me in the first email is not in the sys.path and you have '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip' in it.
I did not use virtual environments often, but I did some quick look. Often there are problems, it seems. Python binary is copied but I am not sure if it copies libpython as well.
By the way, it is a simple question but, did you activate the environment? Can that be a problem?
I'll try to check what might be a problem.

Cheers
Dragan

[Jaime Souza]

Hi Dragan,

The virtual environment is activated as shown by the prompt decoration "(.venv)". I use Pycharm IDE and it activates by default the pre-configured virtual environment.

I am using the recent macOS High Sierra version 10.13.1. And the location of the libpython's are:

$ sudo find / -name libpython*.* | grep libpython
/usr/lib/libpython2.7.dylib
/usr/lib/libpython.dylib
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libpython2.7.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libpython.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.tbd
/Library/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6.dylib
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6.a
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6.dylib
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.a
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib
/Library/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6m.dylib
/Library/Frameworks/Python.framework/Versions/3.6/share/doc/python3.6/examples/Tools/gdb/libpython.py
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.a

Thanks,
Jaime

[ciroki]

Hi Jaime,
Well, what happens in your case It is very interesting. I recalled what is being done during the daetools compilation and remembered the reasons for @rpath:
https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html#//apple_ref/doc/uid/TP40008306-SW1
If libpython3.6m.dylib is in the search path then it should work fine since it is already loaded by the python executable. Can you run otool -L pyCore.so and check the output?
Also, does it work outside the venv? I.e. in the normal installation?

Cheers
Dragan

[Jaime Souza]

Hi Dragan,

The output of otool seams OK:

otool -L pyCore.so
pyCore.so:
        libpyCore.1.dylib (compatibility version 1.7.0, current version 1.7.3)
        @loader_path/../../solibs/Darwin_x86_64/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        @loader_path/../../solibs/Darwin_x86_64/libcdaeConfig-py36.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/../../solibs/Darwin_x86_64/libboost_python3-daetools-py36.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libpython3.6m.dylib (compatibility version 3.6.0, current version 3.6.0)
        @loader_path/../../solibs/Darwin_x86_64/libboost_system-daetools-py36.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/../../solibs/Darwin_x86_64/libboost_thread-daetools-py36.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/../../solibs/Darwin_x86_64/libboost_filesystem-daetools-py36.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/../../solibs/Darwin_x86_64/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.22.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
        @loader_path/../../solibs/Darwin_x86_64/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
(.venv) ~/PycharmProjects/study-daetools/.venv/lib/python3.6/site-packages/daetools-1.8.0-py3.6-macosx-10.6-intel.egg/daetools/pyDAE/Darwin_x86_64_py36(master ✗) 

I was not able to install outside the venv because I have many others projects using the venv architecture. But I have tried the miniconda approach but I now got a "Segmentation Fault: 11" when I run the first daetools tutorial.

Thanks,
Jaime

[Jaime Souza]

Hi Dragan,

I found the solution, but not the reason.

Reading an issue in another project (LD_LIBRARY_PATH and DYLD_LIBRARY_PATH not imported on OS X), I noted that the LD_LIBRARY_PATH was not set in my env. After the following export:

$ export LD_LIBRARY_PATH="/Library/Frameworks/Python.framework/Versions/3.6/lib/:$LD_LIBRARY_PATH" 

It now works fine.

Maybe it is related to "New versions of OS X enable system integrity protection per default." that prohibits something during the daetool installation script.

Regards,

Jaime

[ciroki]

Hi Jaime,

Well, good that you found a solution and it works. On the other hand, it is strange that it is not handled by the python venv. I know that macOS prohibits tampering with the dynamic library paths for security reasons. That's why I do not rely on LD_LIBRARY_PATH and friends but use the @loader_path so that the libraries are always found relative to the extension module being loaded. The only exceptions are libSystem that is always fixed and libpython that is looked for in the runtime. It is python's responsibility to manage it so that the extension modules can load libpython without an additional effort. But, macOS was always somehow different. I think that when one creates a new venv the python binary gets copied to the specified loacation. Why it is not being done for libpython so that it can be found by the extension modules, remains a mistery. I'll check how it behaves in my system (perhaps it is something very recent) and can I do something to prevent it.

Anyway, thanks a lot for reporting it and finding a solution.

Cheers
Dragan