numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] Final release of Numeric is actually 24.1

[Travis O. <oli...@ee...>]

kh...@ce... wrote:

> On Nov 4, 2005, at 7:51, Travis Oliphant wrote:
>
>> I do hope people are encouraged to move toward scipy core,  however.  
>> It is stabilizing
>
>
> Has anyone actually done this for a non-trivial package? What stops  
> me from even trying scipy.core is the large number of incompatible  
> changes, even though they are mostly superficial. 

There is a module called convertcode.py (in scipy/base) that will make 
nearly all of the required changes.   We used it to convert all of scipy 
over and it was pretty effective.

I can understand an installed base may find it more difficult to 
switch.  That's why we are working hard to get scipy core out of beta as 
soon as possible.  Immediately, we are really looking for people who  
are willing to run their code so we can track down remaining problems. 

-Travis

[Numpy-discussion] bus error in check_kurtosis

[Rob M. <ma...@ll...>]

With todays svn versions (newcore 1440, newscipy 1423)

scipy.test(level=1,verbosity=2) dies with a bus error on check_kurtosis

check_basic (scipy.stats.stats.test_stats.test_mean) ... ok
check_ravel (scipy.stats.stats.test_stats.test_mean) ... ok
check_basic (scipy.stats.stats.test_stats.test_median) ... ok
check_basic (scipy.stats.stats.test_stats.test_mode) ... ok
check_kurtosis (scipy.stats.stats.test_stats.test_moments)Bus error


This is on Mac OSX 10.3.9, python 2.4.1, gcc 3.3.

Is this a known problem?
-- 
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
Rob Managan               email managan at llnl.gov
LLNL                      phone: 925-423-0903
P.O. Box 808, L-095       FAX:   925-422-3389
Livermore, CA  94551-0808

Re: [Numpy-discussion] Final release of Numeric is actually 24.1

[<kh...@ce...>]

On Nov 4, 2005, at 7:51, Travis Oliphant wrote:

> I do hope people are encouraged to move toward scipy core, =20
> however.  It is stabilizing

Has anyone actually done this for a non-trivial package? What stops =20
me from even trying scipy.core is the large number of incompatible =20
changes, even though they are mostly superficial. I cannot give up =20
Numeric compatibility because all of my user base is using Numeric, =20
and will likely stick to Numeric at least until scipy.core is out of =20
beta, possibly longer. In the meantime, I would have to maintain two =20
versions of all my code (nearly all of my Python modules are =20
concerned by some of the changes), which is not something I am =20
looking forward to.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L=E9on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
---------------------------------------------------------------------

Re: [Numpy-discussion] numpy 24.x on 64-bit Athlon

[Darren D. <dd...@co...>]

On Monday 07 November 2005 12:03 am, Travis Oliphant wrote:
> >I also get the following error when I run newscipy's test(5,10):
> >
> >bench_random (scipy.fftpack.basic.test_basic.test_fft)
> >                 Fast Fourier Transform
> >=================================================
> >
> >      |    real input     |   complex input
> >
> >-------------------------------------------------
> > size |  scipy  | Numeric |  scipy  | Numeric
> >-------------------------------------------------
> >
> >
> >  100 |    0.10Segmentation fault
>
> Curious.  Did you rebuild scipy completely?

Yes, in fact it was the first build on a new computer. I just removed my 
build/, and site-packages/scipy  and Numeric/ directories, rebuilt and 
reinstalled Numeric 24.1 and newcore/newscipy and still observe the segfault. 
I also tried installing Scipy-0.3.2, which did not have a problem with 
Numeric-24.1.

Darren

Re: [Numpy-discussion] numpy 24.x on 64-bit Athlon

[Travis O. <oli...@ee...>]

Darren Dale wrote:

>I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I 
>get the following error for both packages.
>
>python test.py
>........E....................................
>======================================================================
>ERROR: Test the diagonal function.
>----------------------------------------------------------------------
>Traceback (most recent call last):
>  File "test.py", line 584, in testDiagonal
>    assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]])
>  File "test.py", line 28, in assert_eq
>    assert eq(a,b)
>  File "test.py", line 23, in eq
>    raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" %
>ValueError: sequences have different shapes:
>a(4, 2)=array([[5, 1],
>       [6, 2],
>       [7, 3],
>       [8, 4]])
>b(2, 3)=[[2, 7, 4], [2, 7, 4]]
>  
>
This error is a test problem not worth bothering with.   The (>2d) 
diagonal function was always broken under Numeric.   It was changed a 
while ago.  The test is broken, but this doesn't mean anything is wrong 
with Numeric.

>I also get the following error when I run newscipy's test(5,10):
>
>bench_random (scipy.fftpack.basic.test_basic.test_fft)
>                 Fast Fourier Transform
>=================================================
>      |    real input     |   complex input
>-------------------------------------------------
> size |  scipy  | Numeric |  scipy  | Numeric
>-------------------------------------------------
>  
>
>  100 |    0.10Segmentation fault
>  
>

Curious.  Did you rebuild scipy completely? 

-Travis

Re: [Numpy-discussion] numpy 24.x on 64-bit Athlon

[Darren D. <dd...@co...>]

On Saturday 05 November 2005 07:31 pm, Darren Dale wrote:
> I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system.
[...]
> I also get the following error when I run newscipy's test(5,10):
>
> bench_random (scipy.fftpack.basic.test_basic.test_fft)
>                  Fast Fourier Transform
> =================================================
>
>       |    real input     |   complex input
>
> -------------------------------------------------
>  size |  scipy  | Numeric |  scipy  | Numeric
> -------------------------------------------------
>   100 |    0.10Segmentation fault

I should have noted that I only get the segfault from Scipy with Numeric 24.1.

Darren

[Numpy-discussion] numpy 24.x on 64-bit Athlon

[Darren D. <dd...@co...>]

I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I 
get the following error for both packages.

python test.py
........E....................................
======================================================================
ERROR: Test the diagonal function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 584, in testDiagonal
    assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]])
  File "test.py", line 28, in assert_eq
    assert eq(a,b)
  File "test.py", line 23, in eq
    raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" %
ValueError: sequences have different shapes:
a(4, 2)=array([[5, 1],
       [6, 2],
       [7, 3],
       [8, 4]])
b(2, 3)=[[2, 7, 4], [2, 7, 4]]

I also get the following error when I run newscipy's test(5,10):

bench_random (scipy.fftpack.basic.test_basic.test_fft)
                 Fast Fourier Transform
=================================================
      |    real input     |   complex input
-------------------------------------------------
 size |  scipy  | Numeric |  scipy  | Numeric
-------------------------------------------------
  100 |    0.10Segmentation fault

Has anyone else had a problem building Numeric-24.x on a 64-bit AMD system?

Thanks,
Darren

[Numpy-discussion] question about dotblas and ACML

[Darren D. <dd...@co...>]

I just got a new Athlon 64 bit system. I'm in the process of building 
Numeric-24.1 against the AMD Core Math Library. Does anyone know if it is 
possible to enable the use_dotblas option in customize.py with ACML?

Thanks,
Darren

Re: [Numpy-discussion] Yet another problem in CVS numarray

[Todd M. <jm...@st...>]

This turned out to be a problem with the way numarray handles Numeric's 
multi-segment buffer protocol.   I worked around this by implementing 
David Cooke's __array_struct__ array interface for numarray.  I'm now 
seeing times like these:

numarray-->Numeric array_if: [0.20523881912231445, 0.19724917411804199, 
0.17538094520568848]
numarray-->Numeric fromstring: [0.28353691101074219, 
0.21145009994506836, 0.22541189193725586]
Numeric-->numarray array_if: [0.40105700492858887, 0.34915614128112793, 
0.39094209671020508]
Numeric-->numarray buffer_if: [0.29201602935791016, 0.25055789947509766, 
0.23227095603942871]

As you can see,  the numarray-->Numeric array_if is now faster than 
fromstring() and Numeric-->numarray array_if is now getting close to the 
time for constructing a numarray from a buffer.

Todd

Francesc Altet wrote:

>Hi,
>
>I've detected another problem in CVS numarray when converting
>non-contiguous Numeric objects.
>
>With numarray 1.3.3 the next works:
>
>  
>
>>>>Numeric.__version__
>>>>        
>>>>
>'23.8'
>  
>
>>>>numarray.__version__
>>>>        
>>>>
>'1.3.3'
>  
>
>>>>num=Numeric.array([1,2,3,4],'b')
>>>>num2=num[::2]
>>>>num2.iscontiguous()
>>>>        
>>>>
>0
>  
>
>>>>na=numarray.array(num2)
>>>>na
>>>>        
>>>>
>array([1, 3])
>
>but, with CVS version of numarray:
>
>  
>
>>>>numarray.__version__
>>>>        
>>>>
>'1.4.2'
>  
>
>>>>Numeric.__version__
>>>>        
>>>>
>'24.1'
>  
>
>>>>na=numarray.array([1,2,3,4],'b')
>>>>num=Numeric.array([1,2,3,4],'b')
>>>>num2=num[::2]
>>>>num2.iscontiguous()
>>>>        
>>>>
>0
>  
>
>>>>na=numarray.array(num2)
>>>>        
>>>>
>Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 376, 
>in array
>    a = a.astype(type)
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 863, 
>in astype
>    return self.copy()
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 923, 
>in copy
>    c = _gen.NDArray.copy(self)
>  File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in 
>copy
>    arr._itemsize)
>numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=2 
>buffersize=2
>
>Cheers,
>
>  
>

[Numpy-discussion] How to install SciPy Core

[Shu Li <sh...@gm...>]

Hi,
 I am a little confused about how to install SciPy Core. I am new to SciPy
so I think starting with the new core is a good idea. When I installed the
core package first and then scipy, in the process of installing scipy, it
showed it also installed a lot of stuff to the core directory which made me
worry that part of the new core is overwritten by old core. And when I
installed scipy first, then some words on the web saying some "__init__.py"
file will "break the scipy" certainly worried me because I don't know what
the break here means and why people are not fixing it if it is avoidable.
 Also even when new core(0.4.2 beta) is installed the scipy still complaine=
d
not seeing "Numeric", which added to the confusion. Could anybody offer som=
e
explanation? Thanks a lot!

--
best,
Sam (AKA Shu Li)

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Philip A. <pa...@eo...>]

I'm planning to update my boost numeric wrappers
(http://www.eos.ubc.ca/research/clouds/num_util.html) to scipy_core as
soon as things settle down.  Here's an example of some fortran
program wrapped by boost that shows a couple of nice features:
i) mirroring of python types in C++, with indexing, etc., 
ii) transparent memory management (all increfs and decrefs are handled by
boost shared pointers) and iii) docstrings.  I'd be happy to contribute to a
page with some SWIG and boost examples for similar code fragments.


#define PY_ARRAY_UNIQUE_SYMBOL PyArrayHandle
#include <num_util.h>
#include <iostream> 
#include "boost/scoped_array.hpp"

namespace { const char* rcsid = ("@(#) $Id: thermwrap.C,v 1.2 2005/09/05 22:47:57 phil Exp $:"); };

using namespace std;
namespace py = boost::python;
namespace nbpl = num_util;

typedef int wave_unit;

extern "C" {
  void ptqv_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thsp, 
                   float& thesp);
  void ptpsp_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, 
              float& thpt,float& thes, float& thsp, float& thesp);
  void mccla_(const char* atmos, float* z, float* p, float* t,float* rvden, 
                 float* o3den, float* den, int& strnlength);
  void thinv_(float& p,float& t,float& thsp);
  void theinv_(float& p,float& t, float& qsp, float& thsp, float& thesp);
}


py::dict ptqv(float p, float t, float qv)
{
  float psp,tsp,qsp,thsp,thesp;
  ptqv_(p,t,qv,psp,tsp,qsp,thsp,thesp);
  py::dict result;
  result["psp"]=psp;
  result["tsp"]=tsp;
  result["qsp"]=qsp;
  result["thsp"]=thsp;
  result["thesp"]=thesp;
  return result;
}

py::dict ptpsp(float p, float t, float psp)
{
  float qv,tsp,qsp,thpt,thes,thsp,thesp;
  ptpsp_(p,t,qv,psp,tsp,qsp,thpt,thes,thsp,thesp);
  py::dict result;
  result["qv"]=qv;
  result["tsp"]=tsp;
  result["qsp"]=qsp;
  result["thpt"]=thpt;
  result["thes"]=thes;
  result["thsp"]=thsp;
  result["thesp"]=thesp;
  return result;
}

py::dict thinv(float p, float thsp)
{
  float t;
  thinv_(p,t,thsp);
  py::dict result;
  result["t"]=t;
  return result;
}

py::dict theinv(float p, float t, float the)
{
  //t input is first guess
  float qv,thsp;
  theinv_(p,t,qv,thsp,the);
  py::dict result;
  result["t"]=t;
  result["qsp"]=qv;
  result["thsp"]=thsp;
  return result;
}

py::numeric::array mcclatchey(string atmos)
{
  int strlength=atmos.length();
  int length=33;
  boost::scoped_array<float> z(new float[length]);
  boost::scoped_array<float> p(new float[length]);
  boost::scoped_array<float> t(new float[length]);
  boost::scoped_array<float> rvden(new float[length]);
  boost::scoped_array<float> o3den(new float[length]);
  boost::scoped_array<float> den(new float[length]);
  mccla_(atmos.c_str(), z.get(), p.get(), t.get(), rvden.get(), o3den.get(), den.get(),strlength );
  std::vector<int> dims;
  dims.push_back(6);
  dims.push_back(33);
  py::numeric::array standsound(nbpl::makeNum(dims, PyArray_DOUBLE));
  double* sndPtr=(double*) nbpl::data(standsound);
  int index;
  for(int i=0;i<33;++i){
    index=i;
    sndPtr[index]=z[i];
    index=33 + i;
    sndPtr[index]=p[i];
    index=2*33 + i;
    sndPtr[index]=t[i];
    index=3*33 + i;
    sndPtr[index]=rvden[i];
    index=4*33 + i;
    sndPtr[index]=o3den[i];
    index=5*33 + i;
    sndPtr[index]=den[i];
  }
  return standsound;
}

BOOST_PYTHON_MODULE(thermwrap)
{
  using namespace boost::python;
  import_array();
  numeric::array::set_module_and_type("Numeric", "ArrayType");
  scope().attr("RCSID") = rcsid;
  scope().attr("__doc__") = "wrappers for BettsThermo.f routines: ptqv, ptpsp";
  string docstring("input: pressure (hPa), temp (K), qv (kg/kg)\n");
  docstring += "output psp (hPa), tsp (K), qsp (kg/kg), thsp (K), thesp (K)";
  def("ptqv", ptqv,docstring.c_str());
  docstring="input: pressure (hPa), temp (K), psp (hPa)\n";
  docstring += "output: qv (kg/kg), tsp (K), qsp (kg/kg), thpt (theta(p,t)), thes (K), \n";
  docstring +="thsp (theta(psl,tsl), thesp equiv. potential temp.(theta-e at sl)";
  def("ptpsp",ptpsp,docstring.c_str());
  def("thinv",thinv,"thinv");
  def("theinv",theinv,"theinv");
  def("mcclatchey",mcclatchey,"mcclatchey");
}

[Numpy-discussion] Yet another problem in CVS numarray

[Francesc A. <fa...@ca...>]

Hi,

I've detected another problem in CVS numarray when converting
non-contiguous Numeric objects.

With numarray 1.3.3 the next works:

>>> Numeric.__version__
'23.8'
>>> numarray.__version__
'1.3.3'
>>> num=3DNumeric.array([1,2,3,4],'b')
>>> num2=3Dnum[::2]
>>> num2.iscontiguous()
0
>>> na=3Dnumarray.array(num2)
>>> na
array([1, 3])

but, with CVS version of numarray:

>>> numarray.__version__
'1.4.2'
>>> Numeric.__version__
'24.1'
>>> na=3Dnumarray.array([1,2,3,4],'b')
>>> num=3DNumeric.array([1,2,3,4],'b')
>>> num2=3Dnum[::2]
>>> num2.iscontiguous()
0
>>> na=3Dnumarray.array(num2)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 37=
6,=20
in array
    a =3D a.astype(type)
  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 86=
3,=20
in astype
    return self.copy()
  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 92=
3,=20
in copy
    c =3D _gen.NDArray.copy(self)
  File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in=
=20
copy
    arr._itemsize)
numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=3D2=20
buffersize=3D2

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Paulo J. S. S. <pjs...@im...>]

Chris,

I have written an wrapper to some COIN-OR libraries for Python using
Swig. In this process I have developed some simple typemaps for
numarrays array (floating point arrays). It worked flawlessly.

I haven't changed the code for a while, but you may want to take a look
at it, specially the numarray.i file where I define the numarray
typemaps:

http://www.ime.usp.br/~pjssilva/software.html

Good luck,

Paulo

[Numpy-discussion] Re: Adams to you

[Chrissy L. <ch...@bo...>]

Go
overpay
or your Meddicat
isit our Pharm
ss Sh
od days, Quit 
ing f
ions - v
aExpre
op.
Get additional information - http://starterxlq.presetey.com

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Travis O. <oli...@ee...>]

Stefan van der Walt wrote:

>Hi Chris
>
>On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote:
>  
>
>>Bruce Southey wrote:
>>    
>>
>>>Hi,
>>>I found SWIG extremely to use but it only exposes a function to Python but
>>>not to numpy. Thus it is very slow for matrix functions. So if you want
>>>speed then you will have to deal with the APIs.
>>>      
>>>
>>Yes, but can't I deal with them in writing typemaps, and then let SWIG 
>>do the rest of the work? I think I've seen some examples like this 
>>somewhere, but it's been a while and I need to go digging more.
>>    
>>
>
>I use SWIG and Blitz++ this way, and it works well.  I modified
>Fernando's typemap to work with templates.  See attached (does
>this need to be modified for Numeric3?).
>
>  
>
Only a little bit.  I'll mark the changes

>Stéfan
>  
>
>------------------------------------------------------------------------
>
>// -*- C++ -*-
>
>%{
>#include <blitz/array.h>
>#include <blitz/tinyvec.h>
>#include <Numeric/arrayobject.h>
>  
>
#include "scipy/arrayobject.h"

>%}
>
>namespace blitz {
>
>    template <class T, int N> class Array {
>
>	%typemap(in) Array<T,N> & (Array<T,N> M) {
>	    int T_size = sizeof(T);
>	    
>	    blitz::TinyVector<int,N> shape(0);
>	    blitz::TinyVector<int,N> strides(0);
>	    
>	    int *arr_dimensions = ((PyArrayObject*)$input)->dimensions;
>	    int *arr_strides = ((PyArrayObject*)$input)->strides;
>	    
>	    for (int i = 0; i < N; i++) {
>		shape[i]   = arr_dimensions[i];
>		strides[i] = arr_strides[i]/T_size;
>	    }
>	    
>	    M.reference(blitz::Array<T,N>((T*) ((PyArrayObject*)$input)->data,
>					  shape, strides, blitz::neverDeleteData));
>	    
>	    $1 = &M;
>	}
>
>    };
>}
>
>%template(matrix1d) blitz::Array<double, 1>;
>%template(matrix2d) blitz::Array<double, 2>;
>%template(matrix3d) blitz::Array<double, 3>;
>
>%template(matrix1f) blitz::Array<float, 1>;
>%template(matrix2f) blitz::Array<float, 2>;
>%template(matrix3f) blitz::Array<float, 3>;
>  
>
On 32-bit platforms, this code would work fine.

On 64-bit platforms, you would need to change at least the 
arr_dimensions and arr_strides arrays to be intp  (typedef to an integer 
the size of a pointer on the platform).

-Travis

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Stefan v. d. W. <st...@su...>]

Hi Chris

On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote:
> Bruce Southey wrote:
> >Hi,
> >I found SWIG extremely to use but it only exposes a function to Python but
> >not to numpy. Thus it is very slow for matrix functions. So if you want
> >speed then you will have to deal with the APIs.
> 
> Yes, but can't I deal with them in writing typemaps, and then let SWIG 
> do the rest of the work? I think I've seen some examples like this 
> somewhere, but it's been a while and I need to go digging more.

I use SWIG and Blitz++ this way, and it works well.  I modified
Fernando's typemap to work with templates.  See attached (does
this need to be modified for Numeric3?).

It is easy to create a Blitz matrix from a Numeric Array without
copying data.  Unfortunately, Blitz jealously guards its data
(restricted pointers), so that it is not so easy to do the conversion
in the other direction.  If anyone knows an answer to this problem,
I'd be glad to hear it.

Stéfan

[Numpy-discussion] Final release of Numeric is actually 24.1

[Travis O. <oli...@ee...>]

I made a final release of Numeric (24.1).  This is "really" the last 
release of Numeric ;-)

I did it because of David Cooke's excellent array_protocol enhancement 
of __array_struct__.  I wanted to make sure the final Numeric fully 
supported the array protocol including the "faster version."

I also tracked down a bug today in scipy core that was inherited from 
Numeric.   A part of me wanted to not fix the bug in Numeric so that 
people who need a stable platform will move to scipy core :-)  But the 
better part of me won, and I fixed the problem and made a new Numeric 
release.

I do hope people are encouraged to move toward scipy core, however.  It 
is stabilizing quite rapidly.  All of scipy now builds with the new 
scipy core, and all of (full) scipy's 1300 tests pass for both 32-bit 
and 64-bit systems.   I will be making a release of scipy_core (probably 
called version 0.6 this weekend).  It is still missing finished 
records.py, ma.py, and chararray.py modules (being worked on).   When 
these are available I will make release of scipy core version 1.0

Best regards,

-Travis

[Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[RayS <ra...@bl...>]

Have you looked at ctypes code generator?
http://starship.python.net/crew/theller/ctypes/codegen.html
"The generator converts declarations in C header files into 
executable Python code: enums, structs, unions, function 
declarations, com interfaces, and preprocessor definitions."

It has progressed well in the last several months, in my experience. 
I did some tests a while back that found it compared well with weave etc.
http://sourceforge.net/mailarchive/forum.php?thread_id=7636431&forum_id=24606

Ray Schumacher

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Chris B. <Chr...@no...>]

Bruce Southey wrote:
> Hi,
> I found SWIG extremely to use but it only exposes a function to Python but
> not to numpy. Thus it is very slow for matrix functions. So if you want
> speed then you will have to deal with the APIs.

Yes, but can't I deal with them in writing typemaps, and then let SWIG 
do the rest of the work? I think I've seen some examples like this 
somewhere, but it's been a while and I need to go digging more.

Anyone got one?

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Bruce S. <bso...@gm...>]

Hi,
I found SIWG extremely to use but it only exposes a function to Python but
not to numpy. Thus it is very slow for matrix functions. So if you want
speed then you will have to deal with the APIs.

Regards
Bruce

On 11/3/05, Chris Barker <Chr...@no...> wrote:
>
> Hi all,
>
> I suddenly seem to have the need for working with a bunch of different
> existing C and C++ code, so I'm looking for a way to make it easier. I
> love NumPy, so I really want to use NumPy arrays My needs fall into
> three categories:
>
> 1) Writing custom extension code from scratch:
>
> In the past, I've done this by just using the NumPy API, but it seems
> that I shouldn't have to do all that book keeping myself.
>
> 2) Wrapping existing libraries:
>
> At the moment, I'd like to wrap Richard Shewchuk's triangle:
>
> http://www.cs.cmu.edu/~quake/triangle.html
>
> Which is straight C, as well as the Proj4 map projections library, also
> C, and there may be others.
>
> 3) Wrapping in house code, C & C++, that is under development, but I
> want to be able to use it from Python and C++, and also perhaps to write
> tests for it in Python.
>
> The options I'm looking at are:
>
> Pyrex:
>
> This seems like perhaps the easiest way to write extension code, but it
> doesn't do any automatic wrapping.
>
> Boost::Python:
>
> This looks like a very nice way to write extension modules, and it even
> appears to already include support for NumPy arrays (which ones? is that
> support being maintained, and will it support SciPy.base?)
>
> SWIG:
>
> The has the major advantage of automatically wrapping code. I see this
> as a particular strength for wrapping our in house code that is under
> development. In theory, once I've written a bunch of type maps, i can
> just re-run it whenever the code base changes.
>
>
> For each of these, I'd love to hear what people's experiences have been,
> and it would be great if anyone can point me to samples that are small
> but non-trivial. Other options I should consider would be great too.
>
>
> One more question: Should I use NumPy arrays to pass data back and forth
> between Python and C, or should I use the new array object/protocol
> instead? If so, has anyone developed examples, SWIG type maps, etc for
> this?
>
> Thanks for any feedback,
>
> -Chris
>
>
>
>
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> NOAA/OR&R/HAZMAT (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chr...@no...
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server.
> Download
> it for free - -and be entered to win a 42" plasma tv or your very own
> Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: [Numpy-discussion] More on array protocol

[Todd M. <jm...@st...>]

Francesc Altet wrote:

>Hi,
>
>I'm having problems in using the array protocol in numarray-->Numeric in
>some situations. I'm using the PyBuffer_FromReadWriteMemory C call in
>order to create buffers that will be later used to create numarray
>objects (both NumArray and CharArray) in Python space. The problem is
>that the array protocol implementation is doing badly this conversion.
>
>I was able to create a small example in pure python that reproduces the
>problem:
>
>  
>
>>>>import numarray
>>>>from numarray import memory
>>>>import Numeric
>>>>na=numarray.array([1,2])
>>>>Numeric.array(na)
>>>>        
>>>>
>array([1, 2])
>
>but ...
>
>  
>
>>>>wbuf = memory.writeable_buffer(na)
>>>>nawb = numarray.array(wbuf,type="Int32",shape=(2,))
>>>>nawb
>>>>        
>>>>
>array([1, 2])
>  
>
>>>>Numeric.array(nawb)
>>>>        
>>>>
>array([        2, 135510112])
>
>I'm afraid that the problem should be related with the kind of buffer
>that is attached to the numarray object:
>  
>
I can reproduce garbage here using 1.4.1 but not with CVS.  

>>>>na._data
>>>>        
>>>>
><memory at 0x082577f8 with size:0x00000008 held by object 0x401f10a0
>aliasing object 0x00000000>
>  
>
>>>>nawb._data
>>>>        
>>>>
><read-write buffer for 0x401f10a0, size -1, offset 0 at 0x40572020>
>
>Travis, Todd, are you going to support read-write buffers or should I
>try to move to using the memory module in order to cope with this? For a
>series of reasons I'd like to keep using regular read-write buffers, but
>anyway... I think I'll be able to do the change if absolutely necessary.
>  
>
I pretty sure this problem has been solved...  let me know if you're 
still experiencing problems here.

Todd

Re: [Numpy-discussion] Re: byteswap questions

[Travis O. <oli...@ee...>]

Russell E. Owen wrote:

>Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder 
>if it'll be in scipy.core.
>  
>
No, in Numeric the arrays are always assumed to be in machine byte order 
and so sys.byteorder tells all (Numeric.LittleEndian is a Boolean that 
is also defined).

In scipy core, arr.flags.swapped tells you if the bytes are in swapped 
order.

Then,  sys.byteorder or scipy.LittleEndian indicates the machine order.

Or you use the array_interface descriptor.

is_big_endian = (arr.dtypestr[0]  == '>')

-Travis

[Numpy-discussion] Options for wrapping C and C++ code for use with Numeric

[Chris B. <Chr...@no...>]

Hi all,

I suddenly seem to have the need for working with a bunch of different 
existing C and C++ code, so I'm looking for a way to make it easier. I 
love NumPy, so I really want to use NumPy arrays  My needs fall into 
three categories:

1) Writing custom extension code from scratch:

In the past, I've done this by just using the NumPy API, but it seems 
that I shouldn't have to do all that book keeping myself.

2) Wrapping existing libraries:

At the moment, I'd like to wrap Richard Shewchuk's triangle:

http://www.cs.cmu.edu/~quake/triangle.html

Which is straight C, as well as the Proj4 map projections library, also 
C, and there may be others.

3) Wrapping in house code, C & C++, that is under development, but I 
want to be able to use it from Python and C++, and also perhaps to write 
tests for it in Python.

The options I'm looking at are:

Pyrex:

This seems like perhaps the easiest way to write extension code, but it 
doesn't do any automatic wrapping.

Boost::Python:

This looks like a very nice way to write extension modules, and it even 
appears to already include support for NumPy arrays (which ones? is that 
support being maintained, and will it support SciPy.base?)

SWIG:

The has the major advantage of automatically wrapping code. I see this 
as a particular strength for wrapping our in house code that is under 
development. In theory, once I've written a bunch of type maps, i can 
just re-run it whenever the code base changes.


For each of these, I'd love to hear what people's experiences have been, 
and it would be great if anyone can point me to samples that are small 
but non-trivial. Other options I should consider would be great too.


One more question: Should I use NumPy arrays to pass data back and forth 
between Python and C, or should I use the new array object/protocol 
instead? If so, has anyone developed examples, SWIG type maps, etc for this?

Thanks for any feedback,

-Chris







-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Problem with array assignment

[Todd M. <jm...@st...>]

Francesc Altet wrote:

>Now, let's test the conversion for large objects, with data copy and
>without data copy.
>  
>
I didn't consider that case but was thinking the semantics of the array 
interface would be "buffer-like" and therefore non-copying.   So at the 
moment numarray *never* copies and completely ignores the non-sequence 
array() parameters for the "array interface" case.

>For numarray-->Numeric:
>
>  
>
>>>>t2=timeit.Timer("num=Numeric.array(na,copy=0)","import
>>>>        
>>>>
>Numeric;import numarray; na=numarray.arange(100000)")
>  
>
>>>>t2.repeat(3,100)
>>>>        
>>>>
>[0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516]
>  
>
>>>>t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import
>>>>        
>>>>
>Numeric;import numarray; na=numarray.arange(100000)")
>  
>
>>>>t2_copy.repeat(3,100)
>>>>        
>>>>
>[0.50105500221252441, 0.49400091171264648, 0.49266600608825684]
>
>So it seems like if the data copy is taking place.
>
>  
>
I agree and was to be able to verify this by making buffer()s out of the 
resulting arrays to examine their data addresses.

>For Numeric-->numarray:
>
>  
>
>>>>t4=timeit.Timer("na=numarray.array(num,copy=0)","import
>>>>        
>>>>
>Numeric;import numarray; num=Numeric.arange(100000)")
>  
>
>>>>t4.repeat(3,100)
>>>>        
>>>>
>[0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891]
>  
>
>>>>t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import
>>>>        
>>>>
>Numeric;importnumarray; num=Numeric.arange(100000)")
>  
>
>>>>t4_copy.repeat(3,100)
>>>>        
>>>>
>[0.0063569545745849609, 0.004302978515625, 0.0042738914489746094]
>
>Ooops! the times for conversion with copy and without copy are more or
>less the same. Perhaps the copy is not done? It seems so:
>
>  
>
>>>>num=Numeric.arange(10)
>>>>na=numarray.array(num,copy=0)
>>>>na_copy=numarray.array(num,copy=1)
>>>>na.info()
>>>>        
>>>>
>...
>data pointer: 0x08312280 (DEBUG ONLY)
>  
>
>>>>na_copy.info()
>>>>        
>>>>
>...
>data pointer: 0x08312280 (DEBUG ONLY)
>
>i.e. the same data pointer. So it seems that the Numeric-->numarray is
>not copying the data even in the case that we are asking to do that.
>  
>
Yeah,  the copy=X flag is totally ignored at the moment.

>Other minor things (maybe this is just because you are in the middle of
>refactoring):
>
>  
>
>>>>na._data
>>>>        
>>>>
>array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
>while I would expect:
>
>  
>
>>>>na._data
>>>>        
>>>>
><memory at 0x081e3618 with size:0x00000028 held by object 0x401f06a0
>aliasing object 0x00000000>
>
>Also:
>  
>
This is actually correct I think.  Since a Numeric array supports the 
buffer protocol,  it can be used as a buffer.  The "memory" output shown 
above is the output from one particular kind of buffer;  i.e. it's the 
repr of numarray's memory object.

>  
>
>>>>na
>>>>        
>>>>
>Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line
>930, in __repr__
>    return array_repr(self)
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line
>1660, in array_repr
>    lst = arrayprint.array2string(
>  File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line
>188, in array2string
>    separator, prefix)
>  File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line
>140, in _array2string
>    data = _gen.ravel(a)
>  File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line
>918, in copy
>    c = _gen.NDArray.copy(self)
>  File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724,
>in copy
>    arr._itemsize)
>TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers.
>  
>
For some reason Numeric was returning None for __array_strides__ and 
hence this message.   I fixed array() so that strides are not set when 
__array_strides__ returns None.

With the added complexity of handling the shape, type, and copy 
parameters in array(),   I'm now seeing performance like this from CVS:

numarray-->Numeric array_if: [0.40103912353515625, 0.41119098663330078, 
0.41148614883422852]
numarray-->Numeric fromstring: [0.24139499664306641, 
0.27034401893615723, 0.21657609939575195]
Numeric-->numarray array_if: [0.81009912490844727, 0.79906082153320312, 
0.74358081817626953]
Numeric-->numarray buffer_if: [0.27320504188537598, 0.22041201591491699, 
0.22667884826660156]

Todd

[Numpy-discussion] Re: byteswap questions

[Russell E. O. <ro...@ce...>]

In article <Pin...@ca...>,
 Rick White <rl...@st...> wrote:

> On Wed, 2 Nov 2005, Russell E. Owen wrote:
> 
> > To use documented interfaces (i.e. not arra._byteorder) and to avoid
> > byteswapping the input array, I think I'm going to be stuck doing
> > something like:
> >
> > import sys
> > if sys.byteorder == 'big':
> >    isBigendian = not arr.isbyteswapped()
> > else:
> >    isBigendian = arr.isbyteswapped()
> 
> A simpler version:
> 
> isBigendian = arr.isbyteswapped() != numarray.isBigEndian

Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder 
if it'll be in scipy.core.

Based on stunningly fast responses by Jay T Miller to most of the PRs I 
submitted to sourceforge:

- The byteswap and byteswapped methods do just affect the data and not 
the byteorder flag. The doc string for byteswapped said otherwise and 
that is fixed in the repository. I don't know if the manual has been 
clarified.

- To toggle the byteorder flag, use the togglebyteorder method (duh, I 
wish I'd seen that one).

- An official way to figure out if an array is byteswapped is to use the 
__array_typestr__ method and look at the first character of the returned 
string. "<" means little-endian, ">" means big-endian. This is 
apparently also part of the array interface for scipy.core. I sure hope 
there'll be a more obvious method to call than __array_typestr__ at some 
point.

-- Russell