numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] Interesting timing results

[Sasha <nd...@ma...>]

On 1/17/06, Travis Oliphant <oli...@ie...> wrote:
> ...                        Currently all array scalar math goes through t=
he
> entire ufunc machinery.  This is clearly sub-optimal.   It is the reason
> for the scalarmath module that I've started in the src directory.
> Eventually, we should be able to have scalar math as fast as Python
> scalars.

I have implemented "nonzero", &, | and ^ for scalar bools. (See
http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the
timings:

Before:
> python -m timeit -s "from numpy import bool_; x =3D bool_(0)" "x & x"
100000 loops, best of 3: 3.85 usec per loop

Now:
> python -m timeit -s "from numpy import bool_; x =3D bool_(0)" "x & x"
10000000 loops, best of 3: 0.174 usec per loop

This is close to python bool:
> python -m timeit -s "x =3D bool(0)" "x & x"
10000000 loops, best of 3: 0.142 usec per loop

and faster than python int:
> python -m timeit -s "from numpy import bool_; x =3D 0" "x & x"
10000000 loops, best of 3: 0.199 usec per loop

But it is in fact all within the timing error now.

I did not put it in the scalarmath module for two reasons: first,
scalarmath is not hooked to numpy yet and second because C-API does
not provide access to scalar bools yet.  I have posted a proposal for
C-API changes and did not hear any opposition (well, no support
either), so I will implement that soon.

There are a few issues with the new APIs that I proposed.  First is a
simple one: I proposed to expose boolean scalars as named constants to
Python, the question is
how to call them.

1. True_ and False_
2. true and false

The second issue is whether to add new numbers to _ARRAY_API or add a
separate _ARRAY_SCALAR_API .

I've only optimized scalar-scalar case in binary ops and fall back to
old for everything else.  I would like to optimize scalar-array and
array-scalar cases as well, but I wonder if it is apropriate to make
"(bool_(0) | x) is x" true when x is an array. Alternatively=20
(bool_(0) | x)  can become a new array that shares data with x.

-- sasha

Re: [Numpy-discussion] array vs. matrix for matlab-compatible stuff

[Sven S. <sve...@gm...>]

Travis Oliphant schrieb:

> Yes, some of them do still return arrays.  Matrices are longer lived in
> NumPy then the were in Numeric, for sure, but many functions still
> aren't friendly to matrices and convert all inputs to arrays before
> operation.   Originally, I had the asarray(...) function not convert
> matrices by default, but this is too surprising because matrices change
> the '*' and '**' operators which could make your function not work.
> We should convert all the functions that don't handle matrices so they
> will.  I'd like to see matrices survive longer than they do.   There are
> some functions that try to do that
> 

Thanks very much for this useful information! I totally agree, long live the matrices...
In the ebook you are selling, are things like that (which functions preserve matrix-type)
documented? Those kind of things would be a reason for me to buy it.

cheers,
Sven

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Travis O. <oli...@ie...>]

Francesc Altet wrote:

>http://oprofile.sourceforge.net
>
>It is a nice way to do profiling at C level on Linux machines. Running
>the Paulo benchmarks through oprofile can surely bring some light.
>  
>
I ran the following code snippet (timed under a Timeit instance) through 
the oprofile profiler for both NumPy and Numeric, to look at indexing 
speeds.

op = "b = A[::2,::2]; d = A[1:80,:]"

This is what I found:  overall

   61242 53.9606 /usr/bin/python
    17647 15.5488 /usr/lib/python2.4/site-packages/numpy/core/multiarray.so
    15942 14.0466 /lib/tls/libc-2.3.3.so
     7158  6.3069 /no-vmlinux
     6995  6.1633 /usr/lib/python2.4/site-packages/Numeric/_numpy.so

Showing that more time is spent in NumPy than in Numeric doing indexing...

Here's the breakdown for NumPy
 
samples  %        symbol name
2353     13.3337  PyArray_PyIntAsIntp      # This is also slower --- 
called more often?
2060     11.6734  PyArray_MapIterNew    # This calls fancy_indexing_check.
1980     11.2200  slice_GetIndices
1631      9.2424  parse_index
1149      6.5110  arraymapiter_dealloc       # Interesting this is 
taking so long?
1142      6.4714  array_subscript
1121      6.3524  _IsAligned
1069      6.0577  array_dealloc
780       4.4200  fancy_indexing_check
684       3.8760  PyArray_NewFromDescr
627       3.5530  parse_subindex
538       3.0487  PyArray_DescrFromType
534       3.0260  array_subscript_nice
455       2.5783  _IsContiguous
370       2.0967  _IsFortranContiguous
334       1.8927  slice_coerce_index
294       1.6660  PyArray_UpdateFlags
234       1.3260  anonymous symbol from section .plt
161       0.9123  PyArray_Return
120       0.6800  array_alloc
2         0.0113  PyArray_Broadcast
2         0.0113  PyArray_IterNew
1         0.0057  LONG_setitem
1         0.0057  PyArray_EquivTypes
1         0.0057  PyArray_FromAny
1         0.0057  PyArray_FromStructInterface
1         0.0057  PyArray_IntpConverter
1         0.0057  PyArray_SetNumericOps
1         0.0057  initialize_numeric_types


Here's the breakdown for Numeric:

1577     22.5447  slice_GetIndices
1155     16.5118  parse_index
912      13.0379  PyArray_FromDimsAndDataAndDescr
792      11.3224  array_subscript
675       9.6497  PyArray_IntegerAsInt
517       7.3910  parse_subindex
401       5.7327  array_dealloc
379       5.4182  slice_coerce_index
339       4.8463  array_subscript_nice
161       2.3016  anonymous symbol from section .plt
82        1.1723  PyArray_Return
5         0.0715  do_sliced_copy


Anybody interested in optimization?

-Travis

Re: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2

[Travis O. <oli...@ie...>]

Colin J. Williams wrote:

>>
> Thanks for this, I'm using Numpy 0.9.2.  Does anyone have any 
> suggestion as to what the the other ingredients might be.
> Similar problems arise with Boa-constructer and Python Scripter.  The 
> latter also reports a floating point error.

I'm pretty sure it has to do with the fact that I'm using the mingw32 
compiler to compile NumPy, while Pythonwin was compiled using something 
else. 

It might be related to Py_complex numbers which in Python are passed 
around as structures and my understanding is different compilers treat 
this differently. 

But, it might be something else as well.   I've been surprised in the 
past by some of the quirks of compiling on windows.

The safest thing to do is compile extensions with the same compiler used 
to compile Python (but I don't have that compiler, so I can't do it).

-Travis

Re: [Numpy-discussion] array vs. matrix for matlab-compatible stuff

[Sven S. <sve...@gm...>]

Sven Schreiber schrieb:

> So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards,

Well I hope the fact that the above line is syntactically scandalous and a good reason to feel
ashamed of myself is not the reason that nobody is answering my previous post .... ;-)

But seriously, isn't there a preferred way to do matrix computations in numpy?
Should I stick with arrays a la "linalg.inv(eye(3))", or does it make more sense to use matrices, as
in "mat(eye(3)).I" ? Or, since both notations seem a little clumsy to me, isn't there an even better
way that maybe even combines the best of both worlds? I definitely have the feeling that I'm missing
something.

_Any_ feedback is highly appreciated,
thanks, Sven

Re: [Numpy-discussion] array vs. matrix for matlab-compatible stuff

[Travis O. <oli...@ie...>]

Sven Schreiber wrote:

>I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically
>matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices?
>
>It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm
>glad about short notations like A.I or A*B representing standard matrix operations. Much easier than
>linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem
>to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?).
>So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards,
>which seems clumsy. So what's the best strategy here?
>  
>
Yes, some of them do still return arrays.  Matrices are longer lived in 
NumPy then the were in Numeric, for sure, but many functions still 
aren't friendly to matrices and convert all inputs to arrays before 
operation.   Originally, I had the asarray(...) function not convert 
matrices by default, but this is too surprising because matrices change 
the '*' and '**' operators which could make your function not work. 

We should convert all the functions that don't handle matrices so they 
will.  I'd like to see matrices survive longer than they do.   There are 
some functions that try to do that

-Travis

Re: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2

[Colin J. W. <cj...@sy...>]

Sven Schreiber wrote:

>Colin J. Williams schrieb:
>  
>
>>Example:
>>    
>>
>>>PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)]
>>>on win32.
>>>Portions Copyright 1994-2004 Mark Hammond (mha...@sk...) -
>>>see 'Help/About PythonWin' for further copyright information.
>>> >>> from numpy.core import umath
>>>THIS CRASHES PYTHONWIN
>>>      
>>>
>
>not for me, so there must be some other ingredients as well
>-sven
>
>  
>
Thanks for this, I'm using Numpy 0.9.2.  Does anyone have any suggestion 
as to what the the other ingredients might be.
Similar problems arise with Boa-constructer and Python Scripter.  The 
latter also reports a floating point error.

Colin W.

[Numpy-discussion] include file location and .eggs

[Andrew S. <str...@as...>]

(We've had threads about this before.  See
http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html
and
http://sourceforge.net/mailarchive/forum.php?thread_id=9474853&forum_id=4890)

There's been recent discussion regarding the location of the numpy
include files. The pre-numpy way (Numeric and numarray) was to use
distutils's install_headers, which places them in the long-familiar
locations like /usr/local/include/python2.3/ (on *nix). This was a
solution we all lived with for many years. Why then, is numpy rocking
the boat and putting the include files somewhere else?

For me, and I suspect for most others in favor of the new way, the
answer boils down to using eggs with out-of-the box numpy. Eggs are the
distributable form of Python package distributions made with setuptools
(see http://peak.telecommunity.com/DevCenter/setuptools ). This isn't an
email to convince everyone to use eggs, but I would like to say what
issues eggs solve, what they make possible, and why the current include
file location doesn't work with them. I _would_ like to convince people
that supporting eggs is a valid thing for numpy to do, even if it means
some change in the include file location and therefore some pain.
(Although the amount of pain seems minimal to me.)  I hope the
developers of numpy-based extension modules, even if they never plan on
using eggs, are willing to accomodate those of us who do.

= Why can't eggs use traditional include locations? =
Multiple version support. Using eggs, it is possible to have multiple
versions of the same distribution, such as numpy, installed within a
single Python installation. (I use "distribution" to mean collection of
Python packages and modules created by a call to distutils' setup() and
use that word because some distributions, such as matplotlib, include
multiple top-level packages.) This means I can have a root-installed,
site-wide installation of numpy on my unix machine for all users but I
can easily play with a per-user installed numpy installation if I want
to. With multiple versions of numpy installed on my system, the include
files I need to use to compile an extension module are obviously going
to depend on the right version of the headers. This simply isn't
possible using the traditional include location, but requires the
appropriate headers for the the version of numpy in use. I don't see any
easier solution to this issue than what eggs already has implemented --
import the numpy package and ask it where its include files are (which
will be in a sub-directory of it's top-level space within site-packages).

= What issues do eggs solve? =
I already mentioned multiple version support. eggs also offer dependency
analysis and resolution. They also allow compatibility with
easy_install, meaning automatic downloading and installation of
dependencies (see http://peak.telecommunity.com/DevCenter/EasyInstall ).

= What else do eggs make possible? =
Eggs are distributed as a single ".egg" file, which is really just a
specially formatted .zip file. If the package is "zip-safe", eggs are
installed in site-packages as a single zip file. Even if they're not
zip-safe, eggs are installed in site-packages in their own single
directory. This makes package management much easier.
Another of my favorite enhancements that setuptools brings to Python is
runtime plugin support. If I have package A, which can optionally use
plugins from module B, I can install A without installing B. When I
install B, A will automatically be able to use B. If I write C to the
same plugin interface that B uses, A can also use C, even if A never
knew of the existance of C. There is lots more that setuptools enables.
This is necessarily only a short list of what eggs can do and are useful
for. Please see http://peak.telecommunity.com/DevCenter/setuptools for
more information.

I hope those with reservations about the change in include file
locations now understand the reason for the change and, even more
optimistically, I hope those folks are willing to accept the current
state of affairs. I will attempt to answer any more questions on the issue.

Cheers!
Andrew

Re: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2

[Sven S. <sve...@gm...>]

Colin J. Williams schrieb:
> Example:
>> PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)]
>> on win32.
>> Portions Copyright 1994-2004 Mark Hammond (mha...@sk...) -
>> see 'Help/About PythonWin' for further copyright information.
>>  >>> from numpy.core import umath
>> THIS CRASHES PYTHONWIN
> 

not for me, so there must be some other ingredients as well
-sven

[Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2

[Colin J. W. <cj...@sy...>]

Example:
> PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)]
> on win32.
> Portions Copyright 1994-2004 Mark Hammond (mha...@sk...) -
> see 'Help/About PythonWin' for further copyright information.
>  >>> from numpy.core import umath
> THIS CRASHES PYTHONWIN

Re: [Numpy-discussion] arrayobject.h (and friends) install location

[<kon...@la...>]

On 17.01.2006, at 18:29, Andrew Straw wrote:

> Also, as for as I know, the only packages that install things into
> /usr/include/python2.x/packagename are Numeric and numarray, so I  
> would
> argue it's not "standard", (although it may have lots of history).

Two of my own packages, ScientificPython and MMTK, do the same, so  
that's already four. In fact, I don't know any package that puts its  
include files elsewhere. Most packages have no include files at all  
of course.

Another argument for this being the "standard" location is that it is  
in the include file search path when Python modules are built using  
distutils, and the path where Distutils puts them.. Yet another  
argument is that it's about the only location where include files can  
be placed without any necessity for client packages to know the  
habits of each of its dependencies.


On 18.01.2006, at 02:18, Christopher Barker wrote:

> As a Linux+OS-X user, I've run into a number of these Apple "Think  
> Different" issues, but I don't think creating an ugly, non-standard  
> system to accommodate the fact that Apple like to do things  
> strangely is the right solution. Apple doesn't maintain their  
> python distribution well, or really play well with the python  
> community, so many of us end up installing a new Python anyway.

There isn't much difference between MacOS X and Linux: on both  
systems you need superuser rights to write to /usr/include.


On 18.01.2006, at 03:22, Travis Oliphant wrote:

> This is the important point in this discussion.   There was a  
> reason it was changed.  I remember opposing the change,  
> originally.  However, it solved a real problem and only required  
> people to give up their arbitrary notions of where include files  
> *should* be.  I don't see this as a real issue since you can get  
> where the include files are very easily.

It's hardly arbitrary - see above. And I don't think anyone cares  
where include files are, as long as installing them via distutils  
will put them somewhere where another package installation using  
distutils will find them.

> With that, however, I think it would be helpful for one of those  
> people who use .eggs to clarify why it solves their problem to have  
> the include files in their current location.

Also what their problem exactly is - I don't remember seeing a  
description on this list. Perhaps we can come up with a different  
solution.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
------------------------------------------------------------------------ 
-------

[Numpy-discussion] Updated benchmark

[Paulo J. S. S. <pjs...@im...>]

Hello,

I am sending an update benchmark table below. I have noticed that in my
benchmarck code for numpy I was using transpose(A) instead of
A.transpose() which introduces an extra function call. A very big
penalty for tiny matrices.

Now the transpose operation doesn't look that bad in numpy when compared
to matlab.

There is still something on this table that I can't understand. Why are
the index operations so expensive in domension 50? If you look closely
you'll see that numpy is faster in dimension 5, much slower in dimension
50 (2.6 times slower) and then the gap decreases in dimension 500 (1.6
slower).

Best,

Paulo

Tests    x.T*y   x*y.T     A*x     A*B   A.T*x    half    2in2

Dimension: 5
Array     0.82    0.22    0.18    0.26    0.33    0.90    1.07
Matrix    4.54    1.41    0.56    0.66    0.99    2.87    4.34
NumArr    2.82    0.66    0.61    0.71    7.54    6.61   10.56
Numeri    1.15    0.34    0.28    0.38    0.66    0.60    0.71
Matlab    1.60    0.39    0.31    0.33    0.35    1.13    0.91

Dimension: 50
Array     9.03    1.95    0.48   16.60    0.98    3.89    4.07
Matrix   45.94    3.55    0.90   17.30    1.67    5.93    7.42
NumArr   29.18    2.47    0.90   18.23   11.86    7.87   11.88
Numeri   11.91    2.13    0.59   17.21    9.27    0.93    3.02
Matlab   16.12    1.81    0.99   16.28    0.70    1.46    1.66

Dimension: 500
Array     1.09    8.05    1.77  151.32    1.89    3.63    3.86
Matrix    4.87    8.16    1.82  151.44    1.94    4.22    4.35
NumArr    3.12    8.19    1.80  151.66   19.83    3.94    5.30
Numeri    1.43    8.18    1.81  151.68   17.52    2.94    4.08
Matlab    1.91    5.07    1.89  150.52    1.84    2.27    2.74

Re: [Numpy-discussion] array vs. matrix for matlab-compatible stuff

[Sven S. <sve...@gm...>]

Travis Oliphant schrieb:
> Sven Schreiber wrote:
> 
>> Hi,
>> I've spent a couple of weeks with scipy/numpy and the old-to-new
>> transition; now that the transition is over (?) but some confusion is
>> remaining (on my side) I feel the need to ask a basic question about
>> matlab compatibility in terms of matrix (linear algebra) programming.
>>
>> Take "eye" and "identity" for example; given that "eye" supposedly
>> exists to facilitate transiton from matlab to numpy/scipy (correct?),
>> I expected eye to be/return a matrix.
> 
> 
> Historical is the only reason.  Numeric always returned an array for eye
> not a matrix.  We could return a matrix without difficulty especially if
> we put an eye --> identity transition in convertcode.py
> 

I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically
matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices?

It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm
glad about short notations like A.I or A*B representing standard matrix operations. Much easier than
linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem
to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?).
So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards,
which seems clumsy. So what's the best strategy here?

I'm aware that this is very much a basic/newbie question, so a pointer to the answer(s) would also
be very welcome. If you think I should purchase the numpy-book to get the authorative answer, feel
free to tell me so (but only if it really has the answer ;-)

Thanks for your help,
Sven

Re: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules

[Francesc A. <fa...@ca...>]

A Dijous 19 Gener 2006 06:03, Sasha va escriure:
> +1
>
> I suggest a little more descriptive exception:
>
> PyErr_Format(PyExc_ImportError,
>                      "extension module compiled against version %X of num=
py
> API" ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersion())

I also like this idea. I'd go a step further and add to the above
message something like this:

"...Nertheless, if you know what are you doing and want to allow this
extension to run, set the environment variable FORCE_NUMPY_RUN to 1
(will continue warning you) or 2 (skip all warnings)."

Of course, one should provide some code to check the environment
variable, but this should be not very difficult.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Arnd B. <arn...@we...>]

On Thu, 19 Jan 2006, Francesc Altet wrote:

> A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure:
> > I think the indexing code will be slower because it is more
> > sophisticated than Numeric's.   Basically, it has to check for fancy
> > indexing before defaulting to the old way.   I see this as more of a
> > slow-down than array creation.   It might be possible to improve it ---
> > more eyeballs are always helpful.  But, I'm not sure how at this point.
>
> I'm sure you already know this:
>
> http://oprofile.sourceforge.net
>
> It is a nice way to do profiling at C level on Linux machines. Running
> the Paulo benchmarks through oprofile can surely bring some light.

What might be nice is to use the output of oprofile and
stick it into kcachegrind,
  http://kcachegrind.sourceforge.net/
to get a GUI to the results.

I haven't done this, but according to
  http://linux.com.hk/penguin/man/1/op2calltree.html
the tool is
`op2calltree` - convert OProfile profiling data to KCachegrind calltree format
and further "This utility is part of the KDE Software Development Kit."

Best, Arnd

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Francesc A. <fa...@ca...>]

A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure:
> I think the indexing code will be slower because it is more
> sophisticated than Numeric's.   Basically, it has to check for fancy
> indexing before defaulting to the old way.   I see this as more of a
> slow-down than array creation.   It might be possible to improve it ---
> more eyeballs are always helpful.  But, I'm not sure how at this point.

I'm sure you already know this:

http://oprofile.sourceforge.net

It is a nice way to do profiling at C level on Linux machines. Running
the Paulo benchmarks through oprofile can surely bring some light.

Cheers,

=2D-=20
>0,0<   Francesc Altet =C2=A0 =C2=A0 http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V. =C2=A0=C2=A0Enjoy Data
 "-"

Re: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules

[Sasha <nd...@ma...>]

+1

I suggest a little more descriptive exception:

PyErr_Format(PyExc_ImportError,
                     "extension module compiled against version %X of numpy=
 API"
                     ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersio=
n())

On 1/18/06, Andrew Straw <str...@as...> wrote:
> Travis Oliphant wrote:
>
> >
> > Just in case you missed the note in the middle of another message.
> >
> > The SVN version of NumPy has a few new C-API calls and requires a
> > rebuild of extension modules.  If you get mysterious segfaults all of
> > a sudden, you haven't recompiled...
> >
> > -Travis
> >
>
> Hi Travis,
>
> I have an idea that might prevent people from accidentally getting
> mysterious errors due to mixing extensions compiled against different
> versions of numpy. Basically, the idea is that a version number is
> changed every time the C API changes. This version number is stored in
> each extension module at compile time, and at run time, each module init
> function that calls import_array() will trigger a test against the
> version number of the numpy core.
>
> I've implemented my idea in the included patch, and, as far as I can
> tell, it actually works. This is based on using the NDARRAY_VERSION
> define in arrayobject.h, which I'm completely unsure if this is a
> suitable use for. Furthermore, there's a lot going on that I don't
> understand in this part of the numpy code. Thus, I consider my patch
> only a proof of concept and hope you (or someone) will take this idea,
> perhaps the patch, and run with it. I hope this prevents many headaches
> in the future...
>
> What do you think?
>
>
>

Re: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules

[Andrew S. <str...@as...>]

Travis Oliphant wrote:

>
> Just in case you missed the note in the middle of another message.
>
> The SVN version of NumPy has a few new C-API calls and requires a 
> rebuild of extension modules.  If you get mysterious segfaults all of 
> a sudden, you haven't recompiled...
>
> -Travis
>

Hi Travis,

I have an idea that might prevent people from accidentally getting 
mysterious errors due to mixing extensions compiled against different 
versions of numpy. Basically, the idea is that a version number is 
changed every time the C API changes. This version number is stored in 
each extension module at compile time, and at run time, each module init 
function that calls import_array() will trigger a test against the 
version number of the numpy core.

I've implemented my idea in the included patch, and, as far as I can 
tell, it actually works. This is based on using the NDARRAY_VERSION 
define in arrayobject.h, which I'm completely unsure if this is a 
suitable use for. Furthermore, there's a lot going on that I don't 
understand in this part of the numpy code. Thus, I consider my patch 
only a proof of concept and hope you (or someone) will take this idea, 
perhaps the patch, and run with it. I hope this prevents many headaches 
in the future...

What do you think?

[Numpy-discussion] numpy svn unit test failure

[Christopher F. <ch...@tr...>]

Tonight's svn build of numpy on OS X 10.4 fails the unit test:

In [2]: numpy.test(1,1)
   Found 3 tests for numpy.distutils.misc_util
   Found 10 tests for numpy.core.umath
   Found 8 tests for numpy.lib.arraysetops
   Found 42 tests for numpy.lib.type_check
   Found 9 tests for numpy.lib.twodim_base
   Found 15 tests for numpy.core.multiarray
   Found 25 tests for numpy.core.ma
   Found 6 tests for numpy.core.defmatrix
   Found 3 tests for numpy.lib.getlimits
   Found 33 tests for numpy.lib.function_base
   Found 3 tests for numpy.dft.helper
   Found 6 tests for numpy.core.records
   Found 8 tests for numpy.core.numeric
   Found 4 tests for numpy.lib.index_tricks
   Found 44 tests for numpy.lib.shape_base
   Found 0 tests for __main__
........................................................................ 
........................................E...E........................... 
........................................................................ 
.....
======================================================================
ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_algebra)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/core/tests/test_defmatrix.py", line 111, in check_basic
     assert allclose((mA ** -i).A, B)
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/core/defmatrix.py", line 155, in __pow__
     x = self.I
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/core/defmatrix.py", line 209, in getI
     from numpy.dual import inv
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/dual.py", line 24, in ?
     import scipy.linalg as linpkg
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/__init__.py", line 8, in ?
     from basic import *
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/basic.py", line 17, in ?
     from flinalg import get_flinalg_funcs
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/flinalg.py", line 15, in ?
     from numpy.distutils.misc_util import PostponedException
ImportError: cannot import name PostponedException

======================================================================
ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_properties)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/core/tests/test_defmatrix.py", line 34, in check_basic
     assert allclose(linalg.inv(A), mA.I)
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/core/defmatrix.py", line 209, in getI
     from numpy.dual import inv
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ 
numpy/dual.py", line 24, in ?
     import scipy.linalg as linpkg
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/__init__.py", line 8, in ?
     from basic import *
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/basic.py", line 17, in ?
     from flinalg import get_flinalg_funcs
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ 
scipy/linalg/flinalg.py", line 15, in ?
     from numpy.distutils.misc_util import PostponedException
ImportError: cannot import name PostponedException

----------------------------------------------------------------------
Ran 221 tests in 2.990s

FAILED (errors=2)
Out[2]: <unittest.TextTestRunner object at 0x120aa70>


--
Christopher J. Fonnesbeck

Population Ecologist, Marine Mammal Section
Fish & Wildlife Research Institute (FWC)
St. Petersburg, FL

Adjunct Assistant Professor
Warnell School of Forest Resources
University of Georgia
Athens, GA

T: 727.235.5570
E: chris at trichech.us

[Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules

[Travis O. <oli...@ie...>]

Just in case you missed the note in the middle of another message.

The SVN version of NumPy has a few new C-API calls and requires a 
rebuild of extension modules.  If you get mysterious segfaults all of a 
sudden, you haven't recompiled...

-Travis

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Perry G. <pe...@st...>]

On Jan 18, 2006, at 8:08 PM, Fernando Perez wrote:
>>> Couldn't that last issue be well dealt with by the fact that today's 
>>> numpy is fairly subclassing-friendly? (which, if I remember 
>>> correctly, wasn't quite the case with at least old Numeric).
>> Does that help? You aren't talking about the fast array subclassing 
>> numpy are you? I'm not  sure what you mean here.
>
> What I meant was that by having good subclassing functionality, it's 
> easier to ward off requests for every feature under the sun.  It's 
> much easier to say:
>
> 'this basic object provides a very small, core set of array features 
> where the focus is on raw speed rather than fancy features; if you 
> need extra features, subclass it and add them yourself'
>
> when the subclassing is actually reasonably easy.  Note that I haven't 
> actually used array subclassing myself (haven't needed it), so I may 
> be mistaken in my comments here, it's just an intuition.
>
Yeah, that makes sense. I was confused by the reference to numpy.

Perry

[Numpy-discussion] Re: Passing context in __array__

[Travis O. <oli...@ee...>]

Alexander Belopolsky wrote:

>On 1/18/06, Travis Oliphant <oli...@ee...> wrote:
>  
>
>>This mail got placed in my junk folder so I did not see it until now.
>>Sorry about the confusion...
>>
>>    
>>
>To add to the confusion, I've interpreted your previous e-mail as "go
>ahead" and commited the changes.  Hopefuly your concerns can be
>addressed and I will not have to retract the change.
>
>  
>
That is actually fine.  I don't mind having changes in SVN that we 
discuss and fix.  It's what its for.  As long as it builds, then great...

I went ahead and altered the C-API based on your desired use and changed 
numpy to work with it.

This means that the SVN version of NumPy requires a recompilation of 
extension modules....

Some of my previous comments did not understand how you had implemented 
the routine.  Hopefully what's in SVN now works correctly (it passes 
your tests...)

>Yes, I agree and confessed to this problem in the comments to my code.
> However, I did not want to touch PyArray_FromAny because it is part
>of published C API.  The best solution would be to expose parts of
>PyArray_FromAny in C API and use those in ufunc code.
>  
>
We are pre 1.0 and so I don't have too much problems adding an extra 
context argument to the PyArray_FromAny call :-)

I've also simplified things a bit, too.   I moved array_fromobject to 
PyArray_FromAny and renamed PyArray_FromAny to 
PyArray_CheckFromAny(...)  as it has a couple more checks to ensure 
DEFAULT_FLAGS are set when ENSURECOPY is set and to check for the 
NOTSWAPPED requirement...  Often flags is 0 or does not have these FLAGS 
(or we have a native byte-order descriptor anyway and so the NOTSWAPPED 
requirement is redundant) and so the simpler PyArray_FromAny
can be used.

Most people are still using PyArray_ContiguousFromObject and friends 
anyway and so the change doesn't affect them at all. 

-Travis

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Fernando P. <Fer...@co...>]

Perry Greenfield wrote:
> On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote:

> Really :-). I remember that conversation and wondered if it had 
> something to do with that. (And I remember Paul Dubois talking to me 
> about similar ideas).  I think it is worth trying (and has been I see, 
> though I would have expected perhaps even a greater speed improvement; 
> somehow I think it should not take a lot of time if you don't need all 
> the type, shape and striding flexibility). It just needs someone to do 
> it.

Maybe putting David's code into the sandbox would be a good starting point.

>>>new then either. I have to believe that if you allowed only Float64  
>>>(and perhaps a complex variant) and used other restrictions then it  
>>>would be much faster for small arrays. One would think it would be 
>>>much  easier to implement than Numeric/numarray/numpy... I've always 
>>>thought  that those looking for really fast small array performance 
>>>would be  better served by something like this. But you'd really have 
>>>to fight  off feature creep. ("This almost meets my needs. If it 
>>>could only do  xxx")
>>
>>Couldn't that last issue be well dealt with by the fact that today's 
>>numpy is fairly subclassing-friendly? (which, if I remember correctly, 
>>wasn't quite the case with at least old Numeric).
> 
> 
> Does that help? You aren't talking about the fast array subclassing 
> numpy are you? I'm not  sure what you mean here.

What I meant was that by having good subclassing functionality, it's easier to 
ward off requests for every feature under the sun.  It's much easier to say:

'this basic object provides a very small, core set of array features where the 
focus is on raw speed rather than fancy features; if you need extra features, 
subclass it and add them yourself'

when the subclassing is actually reasonably easy.  Note that I haven't 
actually used array subclassing myself (haven't needed it), so I may be 
mistaken in my comments here, it's just an intuition.

Cheers,

f

Re: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks

[Perry G. <pe...@st...>]

On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote:

> Perry Greenfield wrote:
>> It's not a new idea. I raised it some time ago and I don't think it 
>> was
>
> I wasn't claiming authorship, sorry if it sounded like that :)  In 
> fact, I remember specifically talking with you about this at scipy'03, 
> in the context of small array performance issues for the 
> at-the-time-nascent numarray, and I'm sure similar things have been 
> done many times before.  I've had it floating in my head since I first 
> saw blitz, back in 2001, and blitz probably got it from...  There's 
> nothing really new under the sun ;)
>
Really :-). I remember that conversation and wondered if it had 
something to do with that. (And I remember Paul Dubois talking to me 
about similar ideas).  I think it is worth trying (and has been I see, 
though I would have expected perhaps even a greater speed improvement; 
somehow I think it should not take a lot of time if you don't need all 
the type, shape and striding flexibility). It just needs someone to do 
it.

>> new then either. I have to believe that if you allowed only Float64  
>> (and perhaps a complex variant) and used other restrictions then it  
>> would be much faster for small arrays. One would think it would be 
>> much  easier to implement than Numeric/numarray/numpy... I've always 
>> thought  that those looking for really fast small array performance 
>> would be  better served by something like this. But you'd really have 
>> to fight  off feature creep. ("This almost meets my needs. If it 
>> could only do  xxx")
>
> Couldn't that last issue be well dealt with by the fact that today's 
> numpy is fairly subclassing-friendly? (which, if I remember correctly, 
> wasn't quite the case with at least old Numeric).

Does that help? You aren't talking about the fast array subclassing 
numpy are you? I'm not  sure what you mean here.

Perry

Re: [Numpy-discussion] Efficient way to handle None->nan?

[Sasha <nd...@ma...>]

>>> from numpy.core.ma import masked_values
>>> from numpy import nan
>>> masked_values([1.0,None,2.0],None).filled(nan).astype(float)
array([ 1.        ,         nan,  2.        ])

On 1/18/06, Russell E. Owen <ro...@ce...> wrote:
> We're getting numeric data from a (MySQL) database. We'd like to use
> numarray or NumPy on the resulting data, but some values may be None. Is
> there a fast, efficient way to replace None with NaN? I'd hate to use a
> list comprehension on each data tuple before turning it into an array,
> but I haven't thought of anything else.
>
> numarray.array and numarray.where are both intolerant of None in the
> input data.
>
> -- Russell
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log fi=
les
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=
=3D121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>