From: Michael S. <ms...@co...> - 2014-07-15 17:27:48
|
Hi Brian, the Py_LIMITED_API stuff is pretty similar in ideas to the Tcl STUBS mechanics, which is supported by SWIG since some years. Not sure if one could steal a few ideas from the Tcl SWIG code. Michael Am 15.07.2014 17:40, schrieb Brian Cole: > Hi Thomas, > > To clarify, I think SWIG will require a fair amount of work to be able > to use Py_LIMITED_API. It was even proposed as a Google summer of code > project last year > (https://github.com/swig/swig/wiki/GSoC-2013-ideas#Python_common_ABI). I > had the same experience with it. Then got side tracked onto trying out > -builtin since the project idea lists it as important to use in > conjunction. However, even that we're having problems with. So looks > like we're going to have to do some contributions to SWIG on that front. > > Also, if it helps, you can take a look at our "Quick Start" guide to get > a sense of the user side of > things: http://docs.eyesopen.com/toolkits/quickstart/python/linuxosx.html > > Cheers, > Brian > > From: Thomas Krijnen <t.k...@gm... <mailto:t.k...@gm...>> > Date: Tuesday, July 15, 2014 4:04 AM > To: Brian L Cole <co...@ey... <mailto:co...@ey...>> > Cc: Swig User List <swi...@li... > <mailto:swi...@li...>> > Subject: Re: [Swig-user] Multi-platform module loader (Python) > > Hi Brian, > > Thank you very much for your elaborate and insightful response and > generously provided code. It looks like our intentions are really > similar, although admittedly you have put much more thought into this > than I have at this point. I have just began to scratch the surface of > the intricacies involved with this so I am very grateful for the > insights you provided. > > I had not looked into different compiler versions, rpath and sdist yet. > Also, especially Py_LIMITED_API is a very interesting suggestion. > Currently my wrapper fails to compile with this directive, but at this > point I can't really oversee whether it is due to unused SWIG > boilerplate or due to meaningful generated code. > > You are right, it might be hard to find the right project to contribute > this too, might be overly Python specific for SWIG and the Python folks > might be scared away by the binary nature. > > I will need some time to study this some more. In case I stumble upon > something useful I will post back. Feel free to do the same :) > > Best, > Thomas > > On Sat, Jul 12, 2014 at 2:40 AM, Brian Cole <co...@ey... > <mailto:co...@ey...>> wrote: > > Hi Thomas, > > This is a problem we've struggled with for 10+ years now. We've > largely baked our own solution over that time, though I would like > to share with you what we've done in the hope that more of this will > make it into the open source world. Unfortunately there is little > being done by the Python packaging community to support binary > dependencies. Continuum is making progress with anaconda, but it's > no where near as ubiquitous as the virtualenv/pip/pypi based solutions. > > To that end we ship pip-installable tar-balls, i.e., they contain a > traditional setup.py file. The tar-balls are generated by the > "python setup.py sdist" command and have the following general > structure: > > OpenEye-python2.6-redhat-6-x64-2014.6.6/ > - OpenEye_python2.6_redhat_6_x64.egg-info/ # contains other dist > stuff created by sdist > - MANIFEST.in # contains recursive-include openeye/libs *.so > - setup.py # allows for traditional "python setup.py install" > - openeye/ > -- oechem.py # modules generated by swig that are platform > independent, can be safely un-tarred on top of each other, hacks up > swig_export_helper a bit like you did > -- libs/ > --- __init__.py # exports the OEGetModule function that uses > openeye_platform for finding and loading the proper shared library > --- openeye_platform.py # contains code for determining which > platform we're running on > --- python2.6-redhat-6-x64-g++4.4/ # contains swig generated shared > library module and C++ shared libraries, all linked with -rpath > ${ORIGIN} to be file system independent > --- python2.7-redhat-6-x64-g++4.4/ # different directory names > match the tar-ball name up above > --- python2.7-redhat-7-x64-g++4.8/ > --- # can be any number of these platform specific directories > > This allows our user base to just "import openeye.oechem" with any > need for setting LD_LIBRARY_PATH or even PYTHONPATH provided they're > using virtualenv/pip. The magic occurs inside > openeye/libs/__init__.py and openeye/libs/openeye_platform.py. I've > attached those files, feel free to use any of the code in there. > > The downside of this approach is we have to enumerate each > linux/python/bit-ness/compiler combination. This means we currently > ship 22 separate tar-balls for Linux and OSX alone. Futhermore, > users are required to choose the right one. We're working on a "meta > package" that works in conjunction with a PyPI server, but don't > have a good place to host that. > > Combining them isn't as feasible as each > openeye/libs/pythonx.x-osname-osversion-bitness-compiler directory > is at least 20 MB. 22 binaries of those size would quickly overwhelm > our repository, especially with the frequency that we release new > versions. Python 3.x at least brings us binary compatibility > guarantees that will allow us to start reducing those 22 separate > tar-balls down to a more manageable and maybe mergable number. > Though that will require SWIG to be able to generate code for > Py_LIMITED_API, something we would like to work on. > > Even with Python ABI compatibility, we would still have to worry > about libc and libstdc++ compatibility. Something we have largely > punted on since end-users have no idea what version of libc they > have. They can usually tell which Linux distribution they're > running, hence our naming scheme around the major linux > distributions. Note, we also have the requirement to be able to load > the proper module from a NFS mounted file system of some random node > in a linux cluster. That node could be either RHEL5, RHEL6, or > RHEL7, clusters are often heterogenous. > > Hope this helps guide you. Would be more than willing to figure out > the right open-source project to contribute this to. Perhaps not > right for SWIG, maybe one of Python's numerous packaging utilities. > Or hopefully someone from the Python packaging world stumbles upon > this and finds some pity for those of us who need to support > binaries across many platforms. > > Cheers, > Brian > > > From: Thomas Krijnen <t.k...@gm... <mailto:t.k...@gm...>> > Date: Monday, July 7, 2014 8:03 AM > To: Swig User List <swi...@li... > <mailto:swi...@li...>> > Subject: [Swig-user] Multi-platform module loader (Python) > > Hi all, > > Is there an appropriate way to facilitate loading the appropriate > binary (hence, platform dependent) module based on some information > from the python interpreter (e.g. sys.version_info, sys.maxsize, > platform.system(), platform.architecture()). Is this maybe even > something the community could standardise on? > > Right now I put several binaries in various folders and change the > generated wrapper file, say my_module.py, accordingly [not adapted > for py25 and below, just as an example]: > ~~~ > # Do not make changes to this file unless you know what you are > doing--modify > # the SWIG interface file instead. > > - > + # TK: NB: Changes are made to facilitate loading appropriate > binary for current platform > > from sys import version_info > if version_info >= (2,6,0): > + def get_search_path(): > + import os, platform > + dn = os.path.dirname(__file__) > + if os.path.exists(os.path.join(dn, 'lib')): > + dn = os.path.join(dn, 'lib', platform.system(), > platform.architecture()[0]) > + return dn > def swig_import_helper(): > - from os.path import dirname > import imp > fp = None > try: > - fp, pathname, description = > imp.find_module('_my_module', [dirname(__file__)]) > + fp, pathname, description = > imp.find_module('_my_module', [get_search_path()]) > except ImportError: > import _my_module > return _my_module > ~~~ > > The reason for doing so being that I want my users to simply be able > to say `import my_module` not having to worry about fiddling with > sys.path or convoluting their import logic in another way. And at > the same time provide them with a universal module that, for > example, can be part git/svn repo with users on different platforms. > > Have more people thought about bundling libraries for multiple > platforms? Curious to hear your thoughts. > > PS: I do realize this is a can worms in terms of python versions, > glibc versions, etc. > > Best, > Thomas > > > > > ------------------------------------------------------------------------------ > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > > > > _______________________________________________ > Swig-user mailing list > Swi...@li... > https://lists.sourceforge.net/lists/listinfo/swig-user > -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: ms...@co... Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215 |