Re: [Swig-user] Multi-platform module loader (Python)

[Thomas K. <t.k...@gm...>]

Hi Brian,

Thank you very much for your elaborate and insightful response and
generously provided code. It looks like our intentions are really similar,
although admittedly you have put much more thought into this than I have at
this point. I have just began to scratch the surface of the intricacies
involved with this so I am very grateful for the insights you provided.

I had not looked into different compiler versions, rpath and sdist yet.
Also, especially Py_LIMITED_API is a very interesting suggestion. Currently
my wrapper fails to compile with this directive, but at this point I can't
really oversee whether it is due to unused SWIG boilerplate or due to
meaningful generated code.

You are right, it might be hard to find the right project to contribute
this too, might be overly Python specific for SWIG and the Python folks
might be scared away by the binary nature.

I will need some time to study this some more. In case I stumble upon
something useful I will post back. Feel free to do the same :)

Best,
Thomas

On Sat, Jul 12, 2014 at 2:40 AM, Brian Cole <co...@ey...> wrote:

>  Hi Thomas,
>
>  This is a problem we've struggled with for 10+ years now. We've largely
> baked our own solution over that time, though I would like to share with
> you what we've done in the hope that more of this will make it into the
> open source world. Unfortunately there is little being done by the Python
> packaging community to support binary dependencies. Continuum is making
> progress with anaconda, but it's no where near as ubiquitous as the
> virtualenv/pip/pypi based solutions.
>
>  To that end we ship pip-installable tar-balls, i.e., they contain a
> traditional setup.py file. The tar-balls are generated by the "python
> setup.py sdist" command and have the following general structure:
>
>  OpenEye-python2.6-redhat-6-x64-2014.6.6/
> - OpenEye_python2.6_redhat_6_x64.egg-info/     # contains other dist stuff
> created by sdist
> - MANIFEST.in     # contains recursive-include openeye/libs *.so
> - setup.py     # allows for traditional "python setup.py install"
> - openeye/
> -- oechem.py     # modules generated by swig that are platform
> independent, can be safely un-tarred on top of each other, hacks up
> swig_export_helper a bit like you did
> -- libs/
> --- __init__.py     # exports the OEGetModule function that uses
> openeye_platform for finding and loading the proper shared library
> --- openeye_platform.py    # contains code for determining which platform
> we're running on
> --- python2.6-redhat-6-x64-g++4.4/  # contains swig generated shared
> library module and C++ shared libraries, all linked with -rpath ${ORIGIN}
> to be file system independent
> --- python2.7-redhat-6-x64-g++4.4/  # different directory names match the
> tar-ball name up above
> --- python2.7-redhat-7-x64-g++4.8/
> --- # can be any number of these platform specific directories
>
>  This allows our user base to just "import openeye.oechem" with any need
> for setting LD_LIBRARY_PATH or even PYTHONPATH provided they're using
> virtualenv/pip. The magic occurs inside openeye/libs/__init__.py and
> openeye/libs/openeye_platform.py. I've attached those files, feel free to
> use any of the code in there.
>
>  The downside of this approach is we have to enumerate each
> linux/python/bit-ness/compiler combination. This means we currently ship 22
> separate tar-balls for Linux and OSX alone. Futhermore, users are required
> to choose the right one. We're working on a "meta package" that works in
> conjunction with a PyPI server, but don't have a good place to host that.
>
>  Combining them isn't as feasible as each
> openeye/libs/pythonx.x-osname-osversion-bitness-compiler directory is at
> least 20 MB. 22 binaries of those size would quickly overwhelm our
> repository, especially with the frequency that we release new versions.
> Python 3.x at least brings us binary compatibility guarantees that will
> allow us to start reducing those 22 separate tar-balls down to a more
> manageable and maybe mergable number. Though that will require SWIG to be
> able to generate code for Py_LIMITED_API, something we would like to work
> on.
>
>  Even with Python ABI compatibility, we would still have to worry about
> libc and libstdc++ compatibility. Something we have largely punted on since
> end-users have no idea what version of libc they have. They can usually
> tell which Linux distribution they're running, hence our naming scheme
> around the major linux distributions. Note, we also have the requirement to
> be able to load the proper module from a NFS mounted file system of some
> random node in a linux cluster. That node could be either RHEL5, RHEL6, or
> RHEL7, clusters are often heterogenous.
>
>  Hope this helps guide you. Would be more than willing to figure out the
> right open-source project to contribute this to. Perhaps not right for
> SWIG, maybe one of Python's numerous packaging utilities. Or hopefully
> someone from the Python packaging world stumbles upon this and finds some
> pity for those of us who need to support binaries across many platforms.
>
>  Cheers,
> Brian
>
>
>   From: Thomas Krijnen <t.k...@gm...>
> Date: Monday, July 7, 2014 8:03 AM
> To: Swig User List <swi...@li...>
> Subject: [Swig-user] Multi-platform module loader (Python)
>
>     Hi all,
>
>  Is there an appropriate way to facilitate loading the appropriate binary
> (hence, platform dependent) module based on some information from the
> python interpreter (e.g. sys.version_info, sys.maxsize, platform.system(),
> platform.architecture()). Is this maybe even something the community could
> standardise on?
>
> Right now I put several binaries in various folders and change the
> generated wrapper file, say my_module.py, accordingly [not adapted for py25
> and below, just as an example]:
> ~~~
>   # Do not make changes to this file unless you know what you are
> doing--modify
>   # the SWIG interface file instead.
>
> -
> + # TK: NB: Changes are made to facilitate loading appropriate binary for
> current platform
>
>   from sys import version_info
>   if version_info >= (2,6,0):
> +     def get_search_path():
> +         import os, platform
> +         dn = os.path.dirname(__file__)
> +         if os.path.exists(os.path.join(dn, 'lib')):
> +             dn = os.path.join(dn, 'lib', platform.system(),
> platform.architecture()[0])
> +         return dn
>       def swig_import_helper():
> -         from os.path import dirname
>           import imp
>           fp = None
>           try:
> -             fp, pathname, description = imp.find_module('_my_module',
> [dirname(__file__)])
> +             fp, pathname, description = imp.find_module('_my_module',
> [get_search_path()])
>           except ImportError:
>               import _my_module
>               return _my_module
> ~~~
>
>  The reason for doing so being that I want my users to simply be able to
> say `import my_module` not having to worry about fiddling with sys.path or
> convoluting their import logic in another way. And at the same time provide
> them with a universal module that, for example, can be part git/svn repo
> with users on different platforms.
>
>  Have more people thought about bundling libraries for multiple platforms?
> Curious to hear your thoughts.
>
>  PS: I do realize this is a can worms in terms of python versions, glibc
> versions, etc.
>
> Best,
> Thomas
>