numpy-discussion

[Numpy-discussion] A C++ library and the new array interface: the best approach?

[Fernando P. <Fer...@co...>]

Hi all,

I'd like to ask for a bit of guidance on the best path to follow regarding the 
integration of an existing C++ library with numpy, given the recent 
developments on these topics:

1. the new array interface specification
2. the ctypes discussions
3. pyrex

Our situation is the following: an existing in-house C++ library defines a 
templated class (Tensor), in many ways similar to a numpy array, but with its 
own extra functionality for other things.  There are many functions in this 
code which take or return Tensors.  I now need to interface this library with 
a Python code which is fully numpy-based, and hopefully be able to go back and 
forth between the C++ and python sides without unnecessary copies.

Since the C++ codebase is somewhat large and already exists, and will continue 
to evolve as a standalone C++ system, something tells me I should just bite 
the typemap bullet and SWIG the darn thing.  But I've never written anything 
beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take 
advantage of the new array interface with SWIG.  I read all the docs under

  - http://svn.scipy.org/svn/PEP

  - http://www.scipy.org/BaseArray

  - http://numeric.scipy.org/array_interface.html

  - the pyrex/array interface wiki pages (the site is not responding right 
now, so I can't give a URL)


but my lack of familiarity with all the details of new type creation got me a 
bit lost.  I'm sure the information I need is all there, but right now I don't 
really see the forest with all the leaves in my face.  I've also read the 
various recent threads on ctypes, as well was the one initiated by Joris:

http://article.gmane.org/gmane.comp.python.numeric.general/6296


So I'd like to know if SWIG is really the best way out in this particular case 
(and any advice on taking advantage of the array interface via SWIG would be 
appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
pyrex in other contexts, but I know it doesn't directly support C++.  However, 
since I have access to all the code, perhaps a pure C layer could be used to 
bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
can be an option?

Any advice will be greatly appreciated.

Best,

f

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Travis O. <oli...@ie...>]

Fernando Perez wrote:
> Hi all,
>
> but my lack of familiarity with all the details of new type creation got me a 
> bit lost.  I'm sure the information I need is all there, but right now I don't 
> really see the forest with all the leaves in my face.  I've also read the 
> various recent threads on ctypes, as well was the one initiated by Joris:
>
> http://article.gmane.org/gmane.comp.python.numeric.general/6296
>
>
> So I'd like to know if SWIG is really the best way out in this particular case 
> (and any advice on taking advantage of the array interface via SWIG would be 
> appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
> pyrex in other contexts, but I know it doesn't directly support C++.  However, 
> since I have access to all the code, perhaps a pure C layer could be used to 
> bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
> can be an option?
>   

I don't have a lot of time to respond right now.  But here are some 
thoughts.

I'm not sure I'd use SWIG at this point. 

I'd probably either

a) use ctypes to call the C++ library (if it is compiled as a *shared* 
library) although to be honest I'm not sure if calling a C++ causes any 
grief. 

b) byte-the bullet and use Boost python (which other's who have 
integrated C++ code with Python seem to really like).  I've never 
wrapped my mind around Boost python, though, so I'm not much help there.

c) write a custom Tensor class that either inherits from the ndarray or 
is a mix-in class with the ndarray as a component.

If it were me, I'd probably be doing a combination of a and c.

I've just finished the chapter in my book that covers and compares f2py, 
weave, pyrex and ctypes integration.   I've been waiting on finishing 
this chapter before sending updates.  You should get the update by the 
end of the week.


Perhaps more later.


-Travis



> Any advice will be greatly appreciated.
>
> Best,
>
> f
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>   

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Albert S. <fu...@gm...>]

Hello all

On Mon, 03 Jul 2006, Travis Oliphant wrote:

> Fernando Perez wrote:
> > Hi all,
> >
> > but my lack of familiarity with all the details of new type creation got me a 
> > bit lost.  I'm sure the information I need is all there, but right now I don't 
> > really see the forest with all the leaves in my face.  I've also read the 
> > various recent threads on ctypes, as well was the one initiated by Joris:
> >
> > http://article.gmane.org/gmane.comp.python.numeric.general/6296
> >
> > So I'd like to know if SWIG is really the best way out in this particular case 
> > (and any advice on taking advantage of the array interface via SWIG would be 
> > appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
> > pyrex in other contexts, but I know it doesn't directly support C++.  However, 
> > since I have access to all the code, perhaps a pure C layer could be used to 
> > bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
> > can be an option?
> 
> I don't have a lot of time to respond right now.  But here are some 
> thoughts.
> 
> I'm not sure I'd use SWIG at this point. 

At least for C code, I would definately agree. If you already know SWIG, 
you might want to try it, but I started without any knowledge of SWIG 
and ctypes when I wanted to wrap a pretty basic library, and after a 
week of puzzling over SWIG, I learned ctypes and wrapped the library in 
a matter of hours.

> I'd probably either
> 
> a) use ctypes to call the C++ library (if it is compiled as a *shared* 
> library) although to be honest I'm not sure if calling a C++ causes any 
> grief. 

If I understand correctly, because each C++ compiler potentially mangles 
C++ identifier names in a different way, there isn't a way for ctypes 
to know what to look for in a shared library.
 
> b) byte-the bullet and use Boost python (which other's who have 
> integrated C++ code with Python seem to really like).  I've never 
> wrapped my mind around Boost python, though, so I'm not much help there.

If there is any way for you to "hide" you C++ code behind an extern "C" 
interface, i.e. write some C-style wrappers on top of your C++ code, 
then you can use ctypes. As luck would have it, this is exactly what the 
libsvm authors did already, which made wrapping libsvm with ctypes 
relatively straightforward.

If you want to achieve things like Python classes deriving from C++ 
classes and vice versa, Boost-Python is what you want to look at. But 
as others have noted, prepare to wait for the compiler when wrapping 
anything of even moderate size.

In Boost-Python world there's Pyste that allows you to generate code. I 
think Pyste uses GCC-XML underneath. I played with Pyste for a day or 
two long ago, but at that stage the code it made differed greatly from 
what I wrote when doing Boost-Python wrapping by hand, so it was a bit 
strange.

In general, I got the feeling that Pyste wasn't flexible enough to wrap 
any kind of C++ library. But maybe you're lucky and your C++ code is 
just right to wrap easily with Pyste.

> c) write a custom Tensor class that either inherits from the ndarray or 
> is a mix-in class with the ndarray as a component.

Don't know much about this option.

In summary (keeping in mind that I'm a bit of a ctypes zealot at 
present), I'd try the "C interface on top of C++ with ctypes" approach, 
and build an ndarray subclass on top these C functions.

Regards,

Albert

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Philip A. <pa...@eo...>]

Travis Oliphant writes:

 > b) byte-the bullet and use Boost python (which other's who have 
 > integrated C++ code with Python seem to really like).  I've never 
 > wrapped my mind around Boost python, though, so I'm not much help there.

I've updated my set of boost helper functions to work with
numpy -- there's a Readme at:

http://www.eos.ubc.ca/research/clouds/software/pythonlibs/num_util/num_util_release2
(http://tinyurl.com/eo6fe)

A simple example which shows some of the boost python features (reference
counting, C++ versions of python types, docstrings, exceptions)
is appended below.  When called with python script http://tinyurl.com/zy57a it
produces the output  http://tinyurl.com/fr7xo

Regards, Phil Austin

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Andrew S. <str...@as...>]

Dear Fernando,

A couple of quick thoughts:

A) Despite my love of Pyrex, it probably isn't the tool for the job --
it doesn't play particularly well with C++, and it would require you to
write custom code for every function wrapped.

B) It sounds like you want something that will more-or-less
automatically generate the python interface from the header files. To my
mind, this suggests SWIG or boost. Boost uses some seriously fancy (and
mind-bending) C++, which can either be a good thing or a bad thing, but
it certainly boosts compile time! There are doubtless other ways. For
example, perhaps you could try using gccxml to auto-generate something
like a Python/ctypes wrapper.

C) The core of whatever you do will probably involve writing code to
view a Tensor as an object that has the __array_struct__ interface. That
way numpy/scipy/whatever can use the data with no copying. So that's a
good place to start and whatever you learn there will probably be useful
regardless of whatever tools you end up with. You'll probably also want
the converse function so that you can view objects exposing
__array_struct__ (numpy arrays) as a Tensor.

Anyhow, that's not much, but perhaps it'll help.

Cheers!
Andrew

Fernando Perez wrote:

>Hi all,
>
>I'd like to ask for a bit of guidance on the best path to follow regarding the 
>integration of an existing C++ library with numpy, given the recent 
>developments on these topics:
>
>1. the new array interface specification
>2. the ctypes discussions
>3. pyrex
>
>Our situation is the following: an existing in-house C++ library defines a 
>templated class (Tensor), in many ways similar to a numpy array, but with its 
>own extra functionality for other things.  There are many functions in this 
>code which take or return Tensors.  I now need to interface this library with 
>a Python code which is fully numpy-based, and hopefully be able to go back and 
>forth between the C++ and python sides without unnecessary copies.
>
>Since the C++ codebase is somewhat large and already exists, and will continue 
>to evolve as a standalone C++ system, something tells me I should just bite 
>the typemap bullet and SWIG the darn thing.  But I've never written anything 
>beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take 
>advantage of the new array interface with SWIG.  I read all the docs under
>
>  - http://svn.scipy.org/svn/PEP
>
>  - http://www.scipy.org/BaseArray
>
>  - http://numeric.scipy.org/array_interface.html
>
>  - the pyrex/array interface wiki pages (the site is not responding right 
>now, so I can't give a URL)
>
>
>but my lack of familiarity with all the details of new type creation got me a 
>bit lost.  I'm sure the information I need is all there, but right now I don't 
>really see the forest with all the leaves in my face.  I've also read the 
>various recent threads on ctypes, as well was the one initiated by Joris:
>
>http://article.gmane.org/gmane.comp.python.numeric.general/6296
>
>
>So I'd like to know if SWIG is really the best way out in this particular case 
>(and any advice on taking advantage of the array interface via SWIG would be 
>appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
>pyrex in other contexts, but I know it doesn't directly support C++.  However, 
>since I have access to all the code, perhaps a pure C layer could be used to 
>bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
>can be an option?
>
>Any advice will be greatly appreciated.
>
>Best,
>
>f
>
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>  
>

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Simon A. <sim...@ui...>]

Hi Fernando,

Fernando Perez schrieb:
[...]
> So I'd like to know if SWIG is really the best way out in this particular case 
> (and any advice on taking advantage of the array interface via SWIG would be 
> appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
> pyrex in other contexts, but I know it doesn't directly support C++.  However, 
> since I have access to all the code, perhaps a pure C layer could be used to 
> bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
> can be an option?

I'm not so sure either, whether SWIG is the way to go. If, however, you
decide to give it a try, you could try a little tool I am currently
working on. It is basically a SWIG typemap definition which allows for
easy use of numpy arrays from C++.

In brief, I have defined a C++ template class around numpy's own
PyArrayObject structure which allows for convenient access of the array
data using C++-like techniques and takes care of type checking and the
like. As it adds only non-virtual methods and no fields, the objects are
memory-wise identical to numpy's own array structure and can hence be
passed between C++ and Python without any copying of data or metadata.

The original idea behind was a bit the opposite of your problem. I had
program for a numerical computation, written in Python, and wanted to
rewrite the most performance-critical parts in C++. With my SWIG
typemap, I can now write the C++ part using my template class and wrap
it such that these objects appear to Python as numpy array objects.

For your problem, it might be a way to write a kind of casting functions
that takes one of your tensor objects and transforms it into my numpy
objects. This should be possible with only copying metadata (i.e. fields
containing dimensions and the like) but with leaving the actual data in
place.

My stuff is still a bit work in progress, and so I don't want to post it
here yet, but it may help you to not have to start from scratch, as  the
typemap code would help you to get started and might be easy to adjust
to your needs.

So, if you are interested, send me a mail, and I would appreciate any
comments from you on how to make my tool into seomthing really reusable.

Regards,
  Simon


-- 
+---
| Simon Anders, Dipl. Phys.
| Institut fuer Theoretische Physik, Universitaet Innsbruck, Austria
| Tel. +43-512-507-6207, Fax -2919
| preferred (permanent) e-mail: sa...@fs...

Re: [Numpy-discussion] A C++ library and the new array interface: the best approach?

[Simon B. <si...@ar...>]

On Mon, 03 Jul 2006 16:41:11 -0600
Fernando Perez <Fer...@co...> wrote:

> 
> So I'd like to know if SWIG is really the best way out in this particular case 
> (and any advice on taking advantage of the array interface via SWIG would be 
> appreciated), or if ctypes or pyrex could be used here.  I'm quite happy using 
> pyrex in other contexts, but I know it doesn't directly support C++.  However, 
> since I have access to all the code, perhaps a pure C layer could be used to 
> bridge the C++/pyrex gap.  Or given the recent praise for ctypes, perhaps that 
> can be an option?

Pyrex can handle some C++. Eg. make objects, and call methods.
You will need to search the pyrex email archives for all the tricks to
get this to work.

Simon.