Re: [Teem-users] Teem python wrapping

[Carlos S. <cs...@sc...>]

Sorry for the 3-month delay... I finally got around to trying this, and this 
seems to work OK to me.

Good news: With gccxml 0.9.0 and the trunk version of ctypeslib, I did not 
need the patches to create bindings for the trunk version of teem. I'm not 
sure what has changed, but maybe the bugs were fixed in a different part of 
the code.

However, there is one big omission on the current code generator of ctypeslib: 
the exported global variables are silently dropped. For example, gage.h has 
all sorts of exports like

/* defaultsGage.c */
GAGE_EXPORT const char *gageBiffKey;
GAGE_EXPORT int gageDefVerbose;
GAGE_EXPORT double gageDefGradMagCurvMin;
GAGE_EXPORT int gageDefRenormalize;
GAGE_EXPORT int gageDefCheckIntegrals;
GAGE_EXPORT int gageDefK3Pack;
GAGE_EXPORT double gageDefDefaultSpacing;
GAGE_EXPORT int gageDefCurvNormalSide;
GAGE_EXPORT double gageDefKernelIntegralNearZero;
GAGE_EXPORT int gageDefRequireAllSpacings;
GAGE_EXPORT int gageDefRequireEqualCenters;
GAGE_EXPORT int gageDefDefaultCenter;
GAGE_EXPORT int gageDefStackUse;
GAGE_EXPORT int gageDefStackRenormalize;
GAGE_EXPORT int gageDefOrientationFromSpacing;

The offending file is

http://svn.python.org/projects/ctypes/trunk/ctypeslib/ctypeslib/codegen/codegenerator.py

In particular, the method Generator.Variable ignores all variables that do not 
have an initialization expression, and that is suboptimal for the teem case. 
I'm attaching a patch that solves the problem.

The final tweak on my process was to create the basic teem python bindings as 
teembase.py, and have a small handwritten teem.py file that makes some things 
nicer (for example, gccxml, afaik, uses preprocessed source, so it's blind to 
the nice #defines of teem). I'm attaching this file, as well as the resulting 
autogenerated teembase.py.

Later today I hope to port some of my class library to this new style of 
autogenerated bindings. I imagine it will not be much work, since they look a 
lot like what I had.

The patch to codegenerator.py might be of general interest to ctypeslib. I 
hope to send Thomas Heller an email about it soon.

-carlos

> Hello,
> 
> This email documents how I recently was able to automatically create 
> python wrappings for Teem.  Jorik Blaas has been very helpful in this 
> effort, especially in creating patches.
> 
> I'm hoping that other Teem users will try this out, and give feedback 
> (on this list).  The length of this email doesn't correspond to the 
> difficulty of generate the workings, I swear.
> 
> If people could email me the "teem.py" file that results from all this, 
> executed on their machine, then I'll be able to get a handle on how 
> platform-specific the code is, which would have to be taken into account 
> if the python wrappings are checked back into Teem itself.  Thanks very 
> much if you can help!!
> 
> ******** INTRODUCTION
> 
> The python wrappings are based on ctypes, as well as a ctypes code 
> generator (called "ctypeslib"):
> 
> http://python.net/crew/theller/ctypes
> 
> http://starship.python.net/crew/theller/wiki/CodeGenerator
> http://starship.python.net/crew/theller/ctypes/old/codegen.html
> 
> Ctypes is a standard part of python, by which C functions in a shared 
> library can be called from python.  However, the argument and return 
> types of the functions must be known in order for call stacks and such 
> not to be trashed.  For example, with a minimal header file "test.h":
> 
> -------------------------
> #include <stdio.h>
> #include <stdlib.h>
> 
> typedef struct {
>   int aaa, bbb;
> } tst;
> 
> extern tst *tstNew();
> extern void tstSet(tst *t, int v);
> extern void tstPrint(tst *t);
> -------------------------
> 
> and its implementation in test.c (both attached), compiled into a shared 
> library "libtest.dylib", you can with ctypes do the following in python 
> (test.py attached):
> 
> -------------------------
> import ctypes
> lt = ctypes.CDLL("libtest.dylib", ctypes.RTLD_GLOBAL)
> 
> class tst(ctypes.Structure):
>    _fields_ = [
>     ('aaa', ctypes.c_int),
>     ('bbb', ctypes.c_int),
> ]
> lt.tstNew.restype = ctypes.POINTER(tst)
> lt.tstNew.argtypes = []
> lt.tstSet.restype = None
> lt.tstSet.argtypes = [ctypes.POINTER(tst), ctypes.c_int]
> lt.tstPrint.restype = None
> lt.tstPrint.argtypes = [ctypes.POINTER(tst)]
> 
> # now for fun
> t = lt.tstNew()
> lt.tstSet(t, 90)
> lt.tstPrint(t)
> -------------------------
> 
> which prints the correct result of calling (the C function) 
> tstPrint(t).  The information about argtypes, restype, and the contents 
> of structs is communicated through python to ctypes.  For something the 
> size of Teem, this should be done automatically, based on the header 
> files declaring what's in the shared library.  This is the role of 
> gxxxml and ctypeslib.  Unfortunately ctypeslib is not a standard part of 
> python.
> 
> ******** NON-WORKING ALTERNATIVES
> 
> The challenge has been that, according to its author, "ctypeslib is 
> mostly unmaintained now", and indeed I had some problems using it (fixes 
> described below).  However, as far as I can tell, ctypeslib the only 
> tool of its kind that's even close to working, and I have been looking 
> around (other suggestions welcome!).  The ctypeslib author suggested 
> using a C parser, written entirely in python (!), which is part of pyglet:
> 
> http://www.pyglet.org
> 
> Specifically, the pyglet/tools/wraptypes/wrap.py tool in the SVN 
> checkout does generate ctypes code, but it died on the unrrdu.h header 
> in Teem, probably because of how it uses the "##" operator of the C 
> preprocessor to glue together symbol names.  Another possibility seemed 
> to be pygccxml and/or Py++
> 
> http://www.language-binding.net/pygccxml/pygccxml.html
> http://www.language-binding.net/pyplusplus/pyplusplus.html
> 
> because these use gccxml to parse header files, and produce a python 
> data structure that mirrors the XML parse tree, which could maybe be 
> traversed to generate the ctypes code.  However, the author of these 
> says "I don't think that I can help you right now, may be in future", 
> noting that Py++ does have functionality for C code generation, but I 
> think its based on Boost, which is a completely unnecessary wrinkle I'd 
> prefer to avoid if possible.  So for the time being I'm sticking with 
> gccxml and ctypeslib.
> 
> ******** GETTING THE SOFTWARE
> 
> I'm using Python 2.6 ("python --version"), which I got here:
> 
> http://www.python.org/download/
> 
> although I think anything after 2.5 should work.  I'm using gccxml 
> version 0.9.0 ("gccxml --version"), which I got by CVS as described here:
> 
> http://www.gccxml.org/HTML/Download.html
> 
> and built with cmake, if I remember correctly.  I then used a version of 
> ctypeslib (found by Jorik) which was intended to work with this more 
> recent version of gccxml:
> 
> http://svn.python.org/projects/ctypes/branches/ctypeslib-gccxml-0.9
> 
> But this required some patches to work; the patch file fixes.patch is 
> attached:
> 
>   svn co 
> http://svn.python.org/projects/ctypes/branches/ctypeslib-gccxml-0.9
>   cd ctypeslib-gccxml-0.9
>   patch -p0 < ../fixes.patch
>   cd ..
> 
> ctypeslib is written in python; there's nothing new to compile at this 
> point.
> 
> ******** CREATING PYTHON/TEEM
> 
> First get Teem (starting from whatever directory should contain the 
> "teem" source dir):
> 
> svn co https://teem.svn.sourceforge.net/svnroot/teem/teem/trunk teem
> 
> and compile it as a shared library.  Directions for doing this with 
> Cmake are now here:
> 
> http://teem.sourceforge.net/build.html
> 
> I created my wrappings in a new directory "teem-wrap", where the xml and 
> python output will be.  Two environment variables identify where Teem 
> and ctypeslib are:
> 
> TEEM is where Teem's "make install" put subdirectories "bin", "lib", and 
> "include" (e.g. ~/teem-build)
> CTYPES is the full path of ctypeslib-gccxml-0.9
> 
> First create (in, for example, "teem-wrap") a meta-header for all the 
> Teem header files (in tcsh):
> 
>   mkdir teemincl
>   cp -r ${TEEM}/include/teem teemincl
>   cd teemincl
>   ls -1 teem/*.h | awk '{print "#include <"$1">"}' > all.h
>   cd ..
> 
> Then create teem.xml:
> 
>   setenv PYTHONPATH $CTYPES
>   python $CTYPES/scripts/h2xml.py `pwd`/teemincl/all.h -I teemincl -o 
> teem.xml
> 
> Then create teem.py (I'm using DYLD_LIBRARY_PATH in a mac, use 
> LD_LIBRARY_PATH elsewhere).  Sorry if my mail program wraps these lines!!
> 
>   setenv DYLD_LIBRARY_PATH ${TEEM}/lib
>   python $CTYPES/scripts/xml2py.py teem.xml -llibteem.dylib -o teem.py \
>         -r 
> "(air|hest|biff|nrrd|ell|unrrdu|moss|gage|bane|limn|seek|hoover|echo|ten|dye|mite).*" 
> 
> 
> I got some error messages:
> 
> PACKING FAILED: total alignment (8/64)
>   warnings.warn(message, UserWarning)
> /Users/gk/teem-wrap/ctypeslib-gccxml-0.9/ctypeslib/codegen/codegenerator.py:529: 
> UserWarning: Structure tenFiberSingle: PACKING FAILED: total alignment 
> (8/64)
>   warnings.warn(message, UserWarning)
> /Users/gk/teem-wrap/ctypeslib-gccxml-0.9/ctypeslib/codegen/codegenerator.py:529: 
> UserWarning: Structure tenEMBimodalParm: PACKING FAILED: total alignment 
> (8/64)
>   warnings.warn(message, UserWarning)
> /Users/gk/teem-wrap/ctypeslib-gccxml-0.9/ctypeslib/codegen/codegenerator.py:529: 
> UserWarning: Structure tenGradientParm: PACKING FAILED: total alignment 
> (8/64)
>   warnings.warn(message, UserWarning)
> 
> but it seems most of the teem.py is there and useful.
> 
> ******** USING PYTHON/TEEM
> 
> Still in the "teem-wrap" directory I created a tensor dataset by:
> 
> tend helix -s 40 40 40 -o helix.nrrd
> 
> then I could run "python fun.py" with the following fun.py (also attached)
> 
> --------------------
> import teem
> n = teem.nrrdNew()
> teem.nrrdLoad(n, "helix.nrrd", None)
> fa = teem.nrrdNew()
> teem.tenAnisoVolume(fa, n, teem.tenAniso_FA, 0.5)
> pr = teem.nrrdNew()
> teem.nrrdProject(pr, fa, 0, teem.nrrdMeasureSum, teem.nrrdTypeDefault)
> rng = teem.nrrdRangeNew(teem.airNaN(), teem.airNaN())
> teem.nrrdRangeSet(rng, pr, False)
> print "The range is [%s,%s]" % (rng.contents.min,rng.contents.max)
> qpr = teem.nrrdNew()
> teem.nrrdQuantize(qpr, pr, rng, 8)
> teem.nrrdSave("fa.png", qpr, None)
> --------------------
> 
> which prints out "The range is [1.52352257032e-07,10.0928049088]" and 
> saves out an fa.png (attached).  To me this shows the promise of python 
> wrappings that closely mirror the C API, and points the way towards a 
> Teem "shell" that permits interactive debugging.
> 
> ******** CURRENTLY NOT WORKING
> 
> The "PACKING FAILED" errors should be fixed; its probably due to my 
> hacky fix to how ctypeslib handles gccxml's sruct descriptions.
> 
> The airEnums (e.g. nrrdType, tenAniso) that map between strings and 
> integers are included in the teem.xml gccxml output, but not included in 
> the teem.py wrapping, hopefully this can also be fixed, but it will 
> probably require more hacking of ctypeslib.
> 
> ******** THE FUTURE OF PYTHON/TEEM
> 
> I gotta wear shades!  I intend on using python as the primary way of 
> interactively using and debugging Teem, and using the Python wrapping to 
> integrate Teem into other software projects, so the completeness and 
> utility of the Python bindings is something that I want to keep 
> improving.  Your feedback is important!
> 
> In addition, one could envision an additional layer which maps Teem's 
> painfully-wanna-be-object-oriented function calls into the methods of a 
> set of real objects that play well together.  I'm hoping this can be 
> explored in more detail soon.
> 
> Gordon
> 
> 
> ------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users