numpy-discussion

[Numpy-discussion] incorporating C/C++ code

[Joris De R. <jo...@st...>]

Hi,

For heavy number crunching I would like to include C and/or C++ functions
in my NumPy programs. They should have/give NumPy arrays as input/output. 
On http://www.scipy.org/Topical_Software I find several suggestions to wrap
C/C++ code: SWIG, weave, Pyrex, Instant, ... but it's quite difficult for me
to have an idea which one I can/should use.

So, a few questions:

Any suggestion for which package I should use? Does this heavily depend
for which purpose I want to use it?

Where can I find the docs for Weave? I find several links on the internet
pointing to http://www.scipy.org/documentation/weave for more info,
but there is nothing anymore.

Thanks in advance,
Joris


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

Re: [Numpy-discussion] incorporating C/C++ code

[Rob H. <ro...@ho...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joris De Ridder wrote:
> Hi,
> 
> For heavy number crunching I would like to include C and/or C++ functions
> in my NumPy programs. They should have/give NumPy arrays as input/output. 
> On http://www.scipy.org/Topical_Software I find several suggestions to wrap
> C/C++ code: SWIG, weave, Pyrex, Instant, ... but it's quite difficult for me
> to have an idea which one I can/should use.
> 
> So, a few questions:
> 
> Any suggestion for which package I should use? Does this heavily depend
> for which purpose I want to use it?

Wrapping C/C++ code is only necessary if the C/C++ code is pre-existing.
I have thusfar only incorporated C code into Numeric python programs by
writing the code natively as a python extension.

Any kind of wrapping will carry a penalty. If you write a python
extension in C you have all the flexibility you need.

Rob Hooft

- --
Rob W.W. Hooft  ||  ro...@ho...  ||  http://www.hooft.net/people/rob/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEo/8XH7J/Cv8rb3QRAm40AJ0YoTy653HP0FWmRN4/zuTFruDwUwCfTgrV
4zfSl3GVT8mneL60zzr2zeY=
=JQrM
-----END PGP SIGNATURE-----

Re: [Numpy-discussion] incorporating C/C++ code

[Travis O. <oli...@ie...>]

Joris De Ridder wrote:
> Hi,
>
> For heavy number crunching I would like to include C and/or C++ functions
> in my NumPy programs. They should have/give NumPy arrays as input/output. 
> On http://www.scipy.org/Topical_Software I find several suggestions to wrap
> C/C++ code: SWIG, weave, Pyrex, Instant, ... but it's quite difficult for me
> to have an idea which one I can/should use.
>   
This is my personal preference order:

1) If you can write Fortran code --- do it and use f2py

2) If you have well-encapsulated functions to call then use
    either weave or ctypes (both are very nice).

3) PyRex is a great option for writing a custom extension module
    that needs a lot of capability built in.

At this point I would not use SWIG or Instant.

So, if Fortran is out for you, then install scipy (or install weave 
separately) and start with weave  http://www.scipy.org/Weave

If you can compile your C/C++ functions as a shared-library, then 
check-out ctypes as well.

-Travis










> So, a few questions:
>
> Any suggestion for which package I should use? Does this heavily depend
> for which purpose I want to use it?
>
> Where can I find the docs for Weave? I find several links on the internet
> pointing to http://www.scipy.org/documentation/weave for more info,
> but there is nothing anymore.
>
> Thanks in advance,
> Joris
>
>
> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
>
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>   

Re: [Numpy-discussion] incorporating C/C++ code

[Louis C. <lco...@po...>]

>> For heavy number crunching I would like to include C and/or C++ functions
>> in my NumPy programs. They should have/give NumPy arrays as input/output.
>> On http://www.scipy.org/Topical_Software I find several suggestions to wrap
>> C/C++ code: SWIG, weave, Pyrex, Instant, ... but it's quite difficult for me
>> to have an idea which one I can/should use.
>>
> This is my personal preference order:
>
> 1) If you can write Fortran code --- do it and use f2py
>
> 2) If you have well-encapsulated functions to call then use
>    either weave or ctypes (both are very nice).
>
> 3) PyRex is a great option for writing a custom extension module
>    that needs a lot of capability built in.
>
> At this point I would not use SWIG or Instant.
>
> So, if Fortran is out for you, then install scipy (or install weave
> separately) and start with weave  http://www.scipy.org/Weave


Now since we are on the topic ;)

I was wondering if there where any issues with say using Psyco
with NumPy ?  http://psyco.sourceforge.net/

Then those number crunching code could still be in Python at least.
Anyone have some benchmarks/comments ?

Regards, Louis.

-- 
Louis Cordier <lco...@po...> cell: +27721472305
Point45 Entertainment (Pty) Ltd. http://www.point45.org

Re: [Numpy-discussion] incorporating C/C++ code

[Christopher B. <Chr...@no...>]

Louis Cordier wrote:
>> At this point I would not use SWIG or Instant.

In general, SWIG makes sense if you have a substantial existing library 
that you need access to, and particularly if that library is evolving 
and needs to be used directly from C/C++ code as well.

If you are writing C/C++ code specifically to be used as a python 
extension, pyrex and boost::python are good choices. There was a Numeric 
add-on to boost::python at one point, I don't know if anyone has 
modified it for numpy.

> I was wondering if there where any issues with say using Psyco
> with NumPy ?  http://psyco.sourceforge.net/

Psyco knows nothing of numpy arrays, and thus can only access them as 
generic Python objects -- so it won't help.

A couple years ago, someone wrote a micro-Numeric package that used 
python arrays as the base storage, and ran it with psyco with pretty 
impressive results. What that tells me is that if psyco could be taught 
to understand numpy arrays, (or at least the generic array interface) it 
could work well. It would be a lot of work, however.

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] incorporating C/C++ code

[Philip A. <pa...@eo...>]

Christopher Barker writes:

 > If you are writing C/C++ code specifically to be used as a python 
 > extension, pyrex and boost::python are good choices. There was a Numeric 
 > add-on to boost::python at one point, I don't know if anyone has 
 > modified it for numpy.

Yes, I've been migrating my extensions to numpy and will put up a new
num_util.h version on the site
(http://www.eos.ubc.ca/research/clouds/num_util.html) this weekend
(it's about a 10 line diff).  When I get a chance I'm also planning to
add a page to the scipy wiki so we can see the same extension wrapped
with boost, swig, f2py and pyrex. -- Phil

Re: [Numpy-discussion] incorporating C/C++ code

[Tim H. <tim...@co...>]

Christopher Barker wrote:
> Louis Cordier wrote:
>   
>>> At this point I would not use SWIG or Instant.
>>>       
>
> In general, SWIG makes sense if you have a substantial existing library 
> that you need access to, and particularly if that library is evolving 
> and needs to be used directly from C/C++ code as well.
>
> If you are writing C/C++ code specifically to be used as a python 
> extension, pyrex and boost::python are good choices. There was a Numeric 
> add-on to boost::python at one point, I don't know if anyone has 
> modified it for numpy.
>
>   
>> I was wondering if there where any issues with say using Psyco
>> with NumPy ?  http://psyco.sourceforge.net/
>>     
>
> Psyco knows nothing of numpy arrays, and thus can only access them as 
> generic Python objects -- so it won't help.
>
> A couple years ago, someone wrote a micro-Numeric package that used 
> python arrays as the base storage, and ran it with psyco with pretty 
> impressive results.
That might have been me. At least I have done this at least once. I even 
still have the code lying around if anyone wants to play with it. No 
guarantee that it hasn't succumbed to bit rot though.

>  What that tells me is that if psyco could be taught 
> to understand numpy arrays, (or at least the generic array interface) it 
> could work well. It would be a lot of work, however.
>   
There's another problem as well. Psyco only really knows about 2 things. 
Integers (C longs actually) and python objects (pointers).  Well, I 
guess that it also knows about arrays of integers/objects as well. It 
does not know how to handle floating point numbers directly. In fact, 
the way it handles floating point numbers is to break them into two 
32-bit chunks and store them as two integers. When one needs to operate 
on the float these two integers need to be retrieved, reassembled, 
operated on and then stuck back into two integers again. As a result, 
psyco is never going to be super fast for floating point, even if it 
learned about numeric arrays. In principle, it could learn about floats, 
but it would require a major rejiggering. As I understand it, Armin has 
no plans to do much more with Psyco other than bug fixes, instead 
working on PyPy. However, Psyco technology will likely go into PyPy 
(which I've mostly lost track of), so it's possible that down the road 
fast numeric stuff could be doable in PyPy.


-tim