Hi Brian,

Thank you very much for your elaborate and insightful response and generously provided code. It looks like our intentions are really similar, although admittedly you have put much more thought into this than I have at this point. I have just began to scratch the surface of the intricacies involved with this so I am very grateful for the insights you provided.

I had not looked into different compiler versions, rpath and sdist yet. Also, especially Py_LIMITED_API is a very interesting suggestion. Currently my wrapper fails to compile with this directive, but at this point I can't really oversee whether it is due to unused SWIG boilerplate or due to meaningful generated code.

You are right, it might be hard to find the right project to contribute this too, might be overly Python specific for SWIG and the Python folks might be scared away by the binary nature.

I will need some time to study this some more. In case I stumble upon something useful I will post back. Feel free to do the same :)


On Sat, Jul 12, 2014 at 2:40 AM, Brian Cole <coleb@eyesopen.com> wrote:
Hi Thomas, 

This is a problem we've struggled with for 10+ years now. We've largely baked our own solution over that time, though I would like to share with you what we've done in the hope that more of this will make it into the open source world. Unfortunately there is little being done by the Python packaging community to support binary dependencies. Continuum is making progress with anaconda, but it's no where near as ubiquitous as the virtualenv/pip/pypi based solutions. 

To that end we ship pip-installable tar-balls, i.e., they contain a traditional setup.py file. The tar-balls are generated by the "python setup.py sdist" command and have the following general structure: 

- OpenEye_python2.6_redhat_6_x64.egg-info/     # contains other dist stuff created by sdist
- MANIFEST.in     # contains recursive-include openeye/libs *.so
- setup.py     # allows for traditional "python setup.py install" 
- openeye/
-- oechem.py     # modules generated by swig that are platform independent, can be safely un-tarred on top of each other, hacks up swig_export_helper a bit like you did
-- libs/
--- __init__.py     # exports the OEGetModule function that uses openeye_platform for finding and loading the proper shared library
--- openeye_platform.py    # contains code for determining which platform we're running on
--- python2.6-redhat-6-x64-g++4.4/  # contains swig generated shared library module and C++ shared libraries, all linked with -rpath ${ORIGIN} to be file system independent
--- python2.7-redhat-6-x64-g++4.4/  # different directory names match the tar-ball name up above
--- python2.7-redhat-7-x64-g++4.8/ 
--- # can be any number of these platform specific directories 

This allows our user base to just "import openeye.oechem" with any need for setting LD_LIBRARY_PATH or even PYTHONPATH provided they're using virtualenv/pip. The magic occurs inside openeye/libs/__init__.py and openeye/libs/openeye_platform.py. I've attached those files, feel free to use any of the code in there. 

The downside of this approach is we have to enumerate each linux/python/bit-ness/compiler combination. This means we currently ship 22 separate tar-balls for Linux and OSX alone. Futhermore, users are required to choose the right one. We're working on a "meta package" that works in conjunction with a PyPI server, but don't have a good place to host that. 

Combining them isn't as feasible as each openeye/libs/pythonx.x-osname-osversion-bitness-compiler directory is at least 20 MB. 22 binaries of those size would quickly overwhelm our repository, especially with the frequency that we release new versions. Python 3.x at least brings us binary compatibility guarantees that will allow us to start reducing those 22 separate tar-balls down to a more manageable and maybe mergable number. Though that will require SWIG to be able to generate code for Py_LIMITED_API, something we would like to work on. 

Even with Python ABI compatibility, we would still have to worry about libc and libstdc++ compatibility. Something we have largely punted on since end-users have no idea what version of libc they have. They can usually tell which Linux distribution they're running, hence our naming scheme around the major linux distributions. Note, we also have the requirement to be able to load the proper module from a NFS mounted file system of some random node in a linux cluster. That node could be either RHEL5, RHEL6, or RHEL7, clusters are often heterogenous.  

Hope this helps guide you. Would be more than willing to figure out the right open-source project to contribute this to. Perhaps not right for SWIG, maybe one of Python's numerous packaging utilities. Or hopefully someone from the Python packaging world stumbles upon this and finds some pity for those of us who need to support binaries across many platforms. 


From: Thomas Krijnen <t.krijnen@gmail.com>
Date: Monday, July 7, 2014 8:03 AM
To: Swig User List <swig-user@lists.sourceforge.net>
Subject: [Swig-user] Multi-platform module loader (Python)

Hi all,

Is there an appropriate way to facilitate loading the appropriate binary (hence, platform dependent) module based on some information from the python interpreter (e.g. sys.version_info, sys.maxsize, platform.system(), platform.architecture()). Is this maybe even something the community could standardise on?

Right now I put several binaries in various folders and change the generated wrapper file, say my_module.py, accordingly [not adapted for py25 and below, just as an example]:
  # Do not make changes to this file unless you know what you are doing--modify
  # the SWIG interface file instead.

+ # TK: NB: Changes are made to facilitate loading appropriate binary for current platform
  from sys import version_info
  if version_info >= (2,6,0):
+     def get_search_path():
+         import os, platform
+         dn = os.path.dirname(__file__)
+         if os.path.exists(os.path.join(dn, 'lib')):
+             dn = os.path.join(dn, 'lib', platform.system(), platform.architecture()[0])
+         return dn
      def swig_import_helper():
-         from os.path import dirname
          import imp
          fp = None
-             fp, pathname, description = imp.find_module('_my_module', [dirname(__file__)])
+             fp, pathname, description = imp.find_module('_my_module', [get_search_path()])
          except ImportError:
              import _my_module
              return _my_module

The reason for doing so being that I want my users to simply be able to say `import my_module` not having to worry about fiddling with sys.path or convoluting their import logic in another way. And at the same time provide them with a universal module that, for example, can be part git/svn repo with users on different platforms.

Have more people thought about bundling libraries for multiple platforms? Curious to hear your thoughts.

PS: I do realize this is a can worms in terms of python versions, glibc versions, etc.