numpy-discussion — Discussion list for all users of Numerical Python

Re: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[Todd M. <jm...@st...>]

On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote:
> On Monday 28 June 2004 12:03 pm, Todd Miller wrote:
> > On Mon, 2004-06-28 at 13:59, Chris Barker wrote:
> > > Sebastian Haase wrote:
> > > > BTW, from the profiling/timing I did you can tell that wxPyPlot
> > > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast
> > > > ...
> > >
> > > Actually, it's probably not that fast, if you are timing on
> > > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how
> > > long it takes your program to tell X what to draw, but it may take
> > > longer than that to actually draw it. However, what you are timing is
> > > all the stuff that is effected by numarray/Numeric.
> > >
> > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really
> > > wanted a Numeric native version, but Robin really didn't want an
> > > additional dependency. We discussed on this list a while back whether
> > > you could compile against Numeric, but let people run without it, and
> > > have it all work unless Someone actually used it. What makes that tricky
> > > is that the functions that test whether a PyObject is a Numeric array
> > > are in Numeric... but it could probably be done if you tried hard enough
> > > (maybe include just that function in wxPython...)
> >
> > numarray-1.0 has two macros for dealing with this:  PyArray_Present()
> > and PyArray_isArray(obj).  The former (safely) determines that numarray
> > is installed, while the latter determines that numarray is installed and
> > that obj is a NumArray.  Both macros serve to guard sections of code
> > which make more extensive use of the numarray C-API to keep them from
> > segfaulting when numarray is not installed.  I think this would be easy
> > to do for Numeric as well.
> >
> > One problem is that compiling a "numarray improved" extension requires
> > some portion of the numarray headers.  I refactored the numarray
> > includes so that a relatively simple set of 3 files can be used to
> > support the Numeric compatible interface (for numarray).  These could
> > either be included in core Python (with a successful PEP) or included in
> > interested packages.  This approach adds a small source code burden
> > somewhere,  but eliminates the requirement for users to have numarray
> > installed either to run or compile from source.
> >
> > I'll send out the draft PEP later today.
> >
> > Regards,
> > Todd
> 
> My original question was just this: Does anyone know why numarray is maybe 10 
> times slower that Numeric with that particular code segment 
> (PySequence_GetItem) ?

Well, the short answer is probably: no.

Looking at the numarray sequence protocol benchmarks in
Examples/bench.py, and looking at what wxPython is probably doing
(fetching a 1x2 element array from an Nx2 and then fetching 2 numerical
values from that)... I can't fully nail it down.  My benchmarks show
that numarray is 4x slower for fetching the two element array but only
1.1x slower for the two numbers;  that makes me expect at most 4x
slower.  

Noticing the 50k __del__ calls in your profile,  I eliminated __del__
(breaking numarray) to see if that was the problem;  the ratios changed
to 2.5x slower and 0.9x slower (actually faster) respectively.

The large number of "Check" routines preceding the numarray path (I
count 7 looking at my copy of wxPython) has me a little concerned.  I
think those checks are  more expensive for numarray because it is a new
style class.  I have a hard time imagining a 10x difference overall, 
but I think Python does have to traverse the numarray class hierarchy
rather than do a type pointer comparison so they are more expensive.

Is 10x a measured number or a gut feel?

One last thought:  because the sequence protocol is being used rather
than raw array access, compiling matplotlib for numarray (or not) is not
the issue.

Regards,
Todd

Re: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[Sebastian H. <ha...@ms...>]

On Monday 28 June 2004 12:03 pm, Todd Miller wrote:
> On Mon, 2004-06-28 at 13:59, Chris Barker wrote:
> > Sebastian Haase wrote:
> > > BTW, from the profiling/timing I did you can tell that wxPyPlot
> > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast
> > > ...
> >
> > Actually, it's probably not that fast, if you are timing on
> > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how
> > long it takes your program to tell X what to draw, but it may take
> > longer than that to actually draw it. However, what you are timing is
> > all the stuff that is effected by numarray/Numeric.
> >
> > I worked on part of the wxPython DC.DrawXXXList stuff, and I really
> > wanted a Numeric native version, but Robin really didn't want an
> > additional dependency. We discussed on this list a while back whether
> > you could compile against Numeric, but let people run without it, and
> > have it all work unless Someone actually used it. What makes that tricky
> > is that the functions that test whether a PyObject is a Numeric array
> > are in Numeric... but it could probably be done if you tried hard enough
> > (maybe include just that function in wxPython...)
>
> numarray-1.0 has two macros for dealing with this:  PyArray_Present()
> and PyArray_isArray(obj).  The former (safely) determines that numarray
> is installed, while the latter determines that numarray is installed and
> that obj is a NumArray.  Both macros serve to guard sections of code
> which make more extensive use of the numarray C-API to keep them from
> segfaulting when numarray is not installed.  I think this would be easy
> to do for Numeric as well.
>
> One problem is that compiling a "numarray improved" extension requires
> some portion of the numarray headers.  I refactored the numarray
> includes so that a relatively simple set of 3 files can be used to
> support the Numeric compatible interface (for numarray).  These could
> either be included in core Python (with a successful PEP) or included in
> interested packages.  This approach adds a small source code burden
> somewhere,  but eliminates the requirement for users to have numarray
> installed either to run or compile from source.
>
> I'll send out the draft PEP later today.
>
> Regards,
> Todd

My original question was just this: Does anyone know why numarray is maybe 10 
times slower that Numeric with that particular code segment 
(PySequence_GetItem) ?

- Sebastian

[Numpy-discussion] Numarray header PEP

[Todd M. <jm...@st...>]

Perry and I have written a draft PEP for the inclusion of some of the
numarray headers within the Python distribution.  The headers enable
extension writers to provide better support for numarray without adding
any additional dependencies (from a user's perspective) to their
packages.  For users who have numarray, an "improved extension" could
provide better performance, and for those without numarray, the
extension could work normally using the Sequence protocol.

The PEP is attached.  It is formatted using the docutils package which
can be used to generate HTML or PDF.  Comments and corrections would be
appreciated.

Regards,
Todd

Re: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[Todd M. <jm...@st...>]

On Mon, 2004-06-28 at 13:59, Chris Barker wrote:
> Sebastian Haase wrote:
> > BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots 
> > 25000 data points in 0.1 secs - so it's _really_ fast ...
> 
> Actually, it's probably not that fast, if you are timing on 
> Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how 
> long it takes your program to tell X what to draw, but it may take 
> longer than that to actually draw it. However, what you are timing is 
> all the stuff that is effected by numarray/Numeric.
> 
> I worked on part of the wxPython DC.DrawXXXList stuff, and I really 
> wanted a Numeric native version, but Robin really didn't want an 
> additional dependency. We discussed on this list a while back whether 
> you could compile against Numeric, but let people run without it, and 
> have it all work unless Someone actually used it. What makes that tricky 
> is that the functions that test whether a PyObject is a Numeric array 
> are in Numeric... but it could probably be done if you tried hard enough 
> (maybe include just that function in wxPython...) 

numarray-1.0 has two macros for dealing with this:  PyArray_Present()
and PyArray_isArray(obj).  The former (safely) determines that numarray
is installed, while the latter determines that numarray is installed and
that obj is a NumArray.  Both macros serve to guard sections of code
which make more extensive use of the numarray C-API to keep them from
segfaulting when numarray is not installed.  I think this would be easy
to do for Numeric as well.

One problem is that compiling a "numarray improved" extension requires
some portion of the numarray headers.  I refactored the numarray
includes so that a relatively simple set of 3 files can be used to
support the Numeric compatible interface (for numarray).  These could
either be included in core Python (with a successful PEP) or included in
interested packages.  This approach adds a small source code burden
somewhere,  but eliminates the requirement for users to have numarray
installed either to run or compile from source.

I'll send out the draft PEP later today.

Regards,
Todd

Re: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[Chris B. <Chr...@no...>]

Sebastian Haase wrote:
> BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots 
> 25000 data points in 0.1 secs - so it's _really_ fast ...

Actually, it's probably not that fast, if you are timing on 
Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how 
long it takes your program to tell X what to draw, but it may take 
longer than that to actually draw it. However, what you are timing is 
all the stuff that is effected by numarray/Numeric.

I worked on part of the wxPython DC.DrawXXXList stuff, and I really 
wanted a Numeric native version, but Robin really didn't want an 
additional dependency. We discussed on this list a while back whether 
you could compile against Numeric, but let people run without it, and 
have it all work unless Someone actually used it. What makes that tricky 
is that the functions that test whether a PyObject is a Numeric array 
are in Numeric... but it could probably be done if you tried hard enough 
(maybe include just that function in wxPython...) The Same applies for 
numarray support.

Anyway, as it stands, wxPython DC methods are faster with Lists or 
Tuples of values than Numeric or Numarray arrays. You might try 
converting to a list with numarray.tolist() before making the DC call.

Another option is to write a few specialized DC functions that take 
numarray arrays to draw, but are not included in wxPython. I think you'd 
get it as fast as possible that way. I intend to do that some day. If 
you want to get it started, I'll help.

You could probably get a particularly nice improvement in drawing a lot 
of rectangles, as looping through a N X 4 array of coords directly would 
be much, much faster that using the Sequence API on the whole thing, and 
on each item, and checking at every step what kind of object everything is.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

[Numpy-discussion] Re: [SciPy-user] IRIX problems

[Martin W. <mar...@gm...>]

Hi Pearu,

thanks for your help. The solution is to replace libcomplib.sgimath with the 
newer libscs, which has the missing symbol. Our sysadmin pointed me to that.

Martin.

On Monday 28 June 2004 17:39, Pearu Peterson wrote:
> On Mon, 28 Jun 2004, Martin Wiechert wrote:
> > Hi all,
> >
> > I'm trying to install scipy on our dunno-how-old IRIX server.
> > I've used a lib called complib.sgimath instead of atlas/LAPACK ...
> > Now I get the following error:
> >
> >     import lapack_lite
> > ImportError: 563534:python2.3: rld: Fatal Error: unresolvable symbol
> > in /user/wiechert/lib/python2.3/site-packages/Numeric/lapack_lite.so:
> > zgelsd_
>
> (This is a Numeric installation problem, not scipy.)
>
> > lapack_lite is linked against following libs:
> > ldd lapack_lite.so
> >         libcomplib.sgimath.so  =>        /usr/lib32/libcomplib.sgimath.so
> >         libblas.so  =>   /usr/lib32/libblas.so
> >         libftn.so  =>    /usr/lib32/libftn.so
> >         libm.so  =>      /usr/lib32/libm.so
> >         libc.so.1  =>    /usr/lib32/libc.so.1
> >
> > What can I do?
>
> Try to find out which libraries in your system contain zgelsd_ and use it
> when linking lapack_lite. When that's unsuccessful then I suggest building
> LAPACK libraries yourself, see the relevant section in
>
>   http://www.scipy.org/documentation/buildatlas4scipy.txt
>
> HTH,
> Pearu
>
> _______________________________________________
> SciPy-user mailing list
> Sci...@sc...
> http://www.scipy.net/mailman/listinfo/scipy-user

[Numpy-discussion] Re: numarray equivalent to sign() function

[Scott D. D. <Sco...@Ac...>]

Mike Zingale wrote:

> Is there a replacement?
def sign(x): return min(max(-1, math.frexp(x)[0]*2), 1)

- Scott David Daniels
Sco...@Ac...

Re: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ...

[Todd M. <jm...@st...>]

On Fri, 2004-06-25 at 11:23, Sebastian Haase wrote:
> >
> > > Also this applies to (all?) other functions that have an 'axis' argument.
> > > And further I just found that putting "too many slicings" to an array
> > > also
> > >
> > > gets silently ignored:
> > > >>> b.shape
> > >
> > > (7, 128, 128, 128)
> > >
> > > >>> b[2,2,2,2,3]
> > >
> > > Traceback (most recent call last):
> > >   File "<input>", line 1, in ?
> > > IndexError: too many indices.
> > >
> > > BUT:
> > > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3]
> > >
> > > [[[[ 0.]]]]
> > >
> > >
> > > I find this very confusing !! Is there any good reason not to have the
> > > "IndexError" exception in all cases ?
> >
> > This is also a bug.  Fixed in CVS now.

I'm afraid I spoke too soon here...  this will be fixed but is not yet.

Thanks again for taking the time to bring this to our attention.

Regards,
Todd

[off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[Sebastian H. <ha...@ms...>]

Hi John,
I wanted to try matplotlib a few days ago, but first I had some trouble 
compiling it (my debian still uses gcc 2-95, which doesn't understand some 
'std' namespace/template stuff) - and then it compiled, but segfaulted.
Maybe I didn't get "set NUMERIX" stuff right - how do I know that it actually 
built _and_ uses the wx-backend ?

BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots 
25000 data points in 0.1 secs - so it's _really_ fast ...
So it would be nice to get to the ground of this ...

Thanks for the comment,
Sebastian


On Friday 25 June 2004 02:12 pm, John Hunter wrote:
> >>>>> "Sebastian" == Sebastian Haase <ha...@ms...> writes:
>
>     Sebastian> Hi, The long story is that I'm looking for a good/fast
>     Sebastian> graph plotting programs; so I found WxPyPlot
>     Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html)
>     Sebastian> It uses wxPython and plots 25000 data points (with
>     Sebastian> lines + square markers) in under one second - using
>     Sebastian> Numeric that is.
>
> Not an answer to your question ....
>
> matplotlib has full numarray support (no need to rely on sequence
> API).  You need to set NUMERIX='numarray' in setup.py before building
> it *and* set numerix : numarray in the matplotlib rc file.  If you
> don't do both of these things, your numarray performance will suffer,
> sometimes dramatically.
>
> With this test script
>
>     from matplotlib.matlab import *
>     N = 25000
>     x = rand(N)
>     y = rand(N)
>     scatter(x,y, marker='s')
>     #savefig('test')
>     show()
>
> You can do a scatter plot of squares, on my machine in under a second
> using numarray (wxagg or agg backend).  Some fairly recent changes to
> matplotlib have moved this drawing into extension code, with an approx
> 10x performance boost from older versions.  The latest version on the
> sf site (0.54.2) however, does have these changes.
>
> To plot markers with lines, you would need
>
>   plot(x,y, marker='-s')
>
> instead of scatter.  This is considerably slower (approx 3s on my
> system), mainly because I haven't ported the new fast drawing of
> marker code to the line class.  This is an easy fix, however, and will
> be added in short order.
>
> JDH
>
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by Black Hat Briefings & Training.
> Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
> digital self defense, top technical experts, no vendor pitches,
> unmatched networking opportunities. Visit www.blackhat.com
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] numarray speed - PySequence_GetItem

[John H. <jdh...@ac...>]

>>>>> "Sebastian" == Sebastian Haase <ha...@ms...> writes:

    Sebastian> Hi, The long story is that I'm looking for a good/fast
    Sebastian> graph plotting programs; so I found WxPyPlot
    Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html)
    Sebastian> It uses wxPython and plots 25000 data points (with
    Sebastian> lines + square markers) in under one second - using
    Sebastian> Numeric that is.

Not an answer to your question ....

matplotlib has full numarray support (no need to rely on sequence
API).  You need to set NUMERIX='numarray' in setup.py before building
it *and* set numerix : numarray in the matplotlib rc file.  If you
don't do both of these things, your numarray performance will suffer,
sometimes dramatically.

With this test script

    from matplotlib.matlab import *
    N = 25000
    x = rand(N)
    y = rand(N)
    scatter(x,y, marker='s')
    #savefig('test')
    show()

You can do a scatter plot of squares, on my machine in under a second
using numarray (wxagg or agg backend).  Some fairly recent changes to
matplotlib have moved this drawing into extension code, with an approx
10x performance boost from older versions.  The latest version on the
sf site (0.54.2) however, does have these changes.

To plot markers with lines, you would need

  plot(x,y, marker='-s')

instead of scatter.  This is considerably slower (approx 3s on my
system), mainly because I haven't ported the new fast drawing of
marker code to the line class.  This is an easy fix, however, and will
be added in short order.

JDH

[Numpy-discussion] numarray speed - PySequence_GetItem

[Sebastian H. <ha...@ms...>]

Hi,
The long story is that I'm looking for a good/fast graph plotting programs; so 
I found WxPyPlot (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html)
It uses wxPython and plots 25000 data points (with lines + square markers) in 
under one second - using Numeric that is. 
[the slow line in WxPyPlot is:
dc.DrawLines(self.scaled)
    where self.scaled is an array of shape (25000,2) and type Float64
]

The short story is that numarray takes maybe 10 times as long as Numeric
and I tracked the problem down into the wxPython SWIG typemap where he does 
this:

<code-sniplet  from wxPoint_LIST_helper() in helpers.cpp  from wxPython>
  wxPoint* wxPoint_LIST_helper(PyObject* source, int *count)
  { <snip>
  bool isFast = PyList_Check(source) || PyTuple_Check(source);
  <snip>
  for (x=0; x<*count; x++) {
          // Get an item: try fast way first.
          if (isFast) {
              o = PySequence_Fast_GET_ITEM(source, x);
          }
          else {
              o = PySequence_GetItem(source, x);
              if (o == NULL) {
                  goto error1;
              }
          }
</code-sniplet>


I'm not 100% sure that this is where the problem lies - is there a chance (or 
a known issue) that numarray does  PySequence_GetItem()  slower than 
Numeric ?

I just ran this again using the python profiler and 
I get this w/ numarray:
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    1.140    1.140    1.320    1.320 gdi.py:554(DrawLines)
        1    1.250    1.250    1.520    1.520 gdi.py:792(_DrawRectangleList)
    50230    0.450    0.000    0.450    0.000 numarraycore.py:501(__del__)
and this with Numeric:
        1    0.080    0.080    0.080    0.080 gdi.py:554(DrawLines)
        1    0.090    0.090    0.090    0.090 gdi.py:792(_DrawRectangleList)


Thanks,
Sebastian Haase

Re: [Numpy-discussion] compress function

[Todd M. <jm...@st...>]

On Thu, 2004-06-24 at 08:38, Peter Verveer wrote:
> The documentation of the compress function states that the condition 
> must be equal to the given axis of the array that is compressed. e.g.:
> 
>  >>> a = array([[1,2],[3,4]])
>  >>> print compress([1,0], a, axis = 1)
> [[1]
>   [3]]
> 
> However, this also works fine:
> 
>  >>> print compress([[1,0],[0,1]], a)
> [1, 4]
> 
> which is great (I need that) but not documented. Is that behaviour 
> intended? 

It is most likely accidental,  but great is great.

> If so it maybe should be documented.

I filed a bug report but I'm not sure when I'll be *able* to do it.

Regards,
Todd

Re: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ...

[Sebastian H. <ha...@ms...>]

On Friday 25 June 2004 05:48 am, Todd Miller wrote:
> On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote:
> > Hi,
> >
> > please take a look at this:
> > >>> na.sum( na.zeros((2,6)) )
> >
> > [0 0 0 0 0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , 0)
> >
> > [0 0 0 0 0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , 1)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , 2)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , 3)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , 4)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , -1)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , -2)
> >
> > [0 0 0 0 0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , -3)
> >
> > [0 0]
> >
> > >>> na.sum( na.zeros((2,6)) , -4)
> >
> > [0 0]
> >
> >
> > I think here should be a ValueError exception thrown rather than
> > defaulting to the '-1'-axis.  Comments ?
>
> This is a bug.  Hopefully I'll get it today.
>
> > Also this applies to (all?) other functions that have an 'axis' argument.
> > And further I just found that putting "too many slicings" to an array
> > also
> >
> > gets silently ignored:
> > >>> b.shape
> >
> > (7, 128, 128, 128)
> >
> > >>> b[2,2,2,2,3]
> >
> > Traceback (most recent call last):
> >   File "<input>", line 1, in ?
> > IndexError: too many indices.
> >
> > BUT:
> > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3]
> >
> > [[[[ 0.]]]]
> >
> >
> > I find this very confusing !! Is there any good reason not to have the
> > "IndexError" exception in all cases ?
>
> This is also a bug.  Fixed in CVS now.
>
> Thanks for the report!
>
> Regards,
> Todd

Thanks so much - I got worried there for a moment ;-)

Regards,
Sebastian

Re: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ...

[Todd M. <jm...@st...>]

On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote:
> Hi,
> please take a look at this:
> >>> na.sum( na.zeros((2,6)) )
> [0 0 0 0 0 0]
> >>> na.sum( na.zeros((2,6)) , 0)
> [0 0 0 0 0 0]
> >>> na.sum( na.zeros((2,6)) , 1)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , 2)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , 3)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , 4)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , -1)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , -2)
> [0 0 0 0 0 0]
> >>> na.sum( na.zeros((2,6)) , -3)
> [0 0]
> >>> na.sum( na.zeros((2,6)) , -4)
> [0 0]
> >>> 
> 
> I think here should be a ValueError exception thrown rather than defaulting to 
> the '-1'-axis.  Comments ?

This is a bug.  Hopefully I'll get it today.

> Also this applies to (all?) other functions that have an 'axis' argument.
> And further I just found that putting "too many slicings" to an array also 
> gets silently ignored:
> >>> b.shape
> (7, 128, 128, 128)
> >>> b[2,2,2,2,3]
> Traceback (most recent call last):
>   File "<input>", line 1, in ?
> IndexError: too many indices.
> 
> BUT:
> >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3]
> [[[[ 0.]]]]
> 
> 
> I find this very confusing !! Is there any good reason not to have the 
> "IndexError" exception in all cases ?

This is also a bug.  Fixed in CVS now.

Thanks for the report!

Regards,
Todd

Re: [Numpy-discussion] ObjectArray paramter order poll

[Francesc A. <fa...@py...>]

A Dijous 24 Juny 2004 20:16, Todd Miller va escriure:
> A while back Colin Williams suggested that we change the parameter order
> of numarray.objects.ObjectArray from:
> 
> def __init__(self, shape=None, objects=None, rank=None)
> 
> to:
> 
> def __init__(self, objects=None, shape=None, rank=None)
> 
> The new order emphasizes building object arrays from existing sequences,
> while the old order emphasizes building empty (full of None) object
> arrays of a specific shape.  Since array() and fromlist() already exist
> to build from sequences,  I thought it was fine to keep the order as is,
> plus I hate breaking s/w interfaces unless it was my idea. :-)
> 
> Opinions please?  

+1 for the second option

-- 
Francesc Alted

Re: [Numpy-discussion] comparing string array with None raises ValueError

[David M. C. <co...@ph...>]

On June 24, 2004 04:10 pm, Todd Miller wrote:
> On Thu, 2004-06-24 at 15:26, David M. Cooke wrote:
> > On June 24, 2004 01:46 pm, Sebastian Haase wrote:
> > > In general it must be OK to compare anything with None, right ?
> > > (BTW, I get the same error with == and !=)
> >
> > No! Not in general!
>
> Well,  this is a good point.  I think the current numerical behavior was
> a hack I stuck in for people who might not be aware of "is".  It's
> looking like a mistake now.

I've fixed up all uses of == None and != None in my copy of the numarray CVS, 
but the anonymous CVS on SourceForge is timing out :-(, so I haven't made a 
patch yet. Basically, I ran 'grep -R "[!=]= *None" *' to find all uses of == 
and != None, and replaced them with is/is not None. That would remove 
possible bugs.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] comparing string array with None raises ValueError

[Todd M. <jm...@st...>]

On Thu, 2004-06-24 at 16:38, Sebastian Haase wrote:
> On Thursday 24 June 2004 01:10 pm, Todd Miller wrote:
> > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote:
> > > On June 24, 2004 01:46 pm, Sebastian Haase wrote:
> > > > In general it must be OK to compare anything with None, right ?
> > > > (BTW, I get the same error with == and !=)
> > >
> > > No! Not in general!
> >
> > Well,  this is a good point.  I think the current numerical behavior was
> > a hack I stuck in for people who might not be aware of "is".  It's
> > looking like a mistake now.
> >
> > > I learnt this back when Numeric implemented rich
> > > comparisions; suddenly, lots of my code broke. You don't actually want
> > > "is this object _equal_ (or not equal) to None", you want "is this object
> > > None", as None is a singleton.
> >
> > However,  given the context of the original question, Sebastian's code
> >
> > doesn't read like *that* kind of None comparison:
> > > > > > type '10a80' - that is, an array of 10 80 char 'strings' :
> > > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field
> > > > > > >>> q.Mrc.hdr('title') != None
> >
> > q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not
> > None.  What did you want it to do Sebastian?
> 
> q.Mrc.hdr('title') is rather an array of 10 'character array'

I thought it was a single nrows x 10 CharArray with 80 character
elements.

>  .. that's 
> different, isn't it ?

It would be.  What does q.Mrc.hdr('title').info() say?

> In any case, I actually already changed my code to the check of 'is None' - 
> which is probably what I wanted anyway. 
> So, the only argument left, is that the original error/exception message I got 
> was kind of "ugly" ;-) And I think/thought if 'something' cannot be 
> _converted_ to a string array it can't _be_  '==' a string array. That's just 
> why I expected the simple result to be 'False'.
> But now I understand that if comparing an array to a scalar (maybe 'None') you 
> always have to decide if this actually compares "the whole array" with the 
> scalar, or if the comparison get "put through" and done on each element 
> "separately" ...
> which it seems is already decided at least in the numerical array case.

I'm not really sure that the numerical array None hack should be
replicated.  Barring an edict from Perry or Rick or an outcry from the
community,  I think I'm going to just leave this alone.

Regards,
Todd

Re: [Numpy-discussion] comparing string array with None raises ValueError

[Sebastian H. <ha...@ms...>]

On Thursday 24 June 2004 01:10 pm, Todd Miller wrote:
> On Thu, 2004-06-24 at 15:26, David M. Cooke wrote:
> > On June 24, 2004 01:46 pm, Sebastian Haase wrote:
> > > In general it must be OK to compare anything with None, right ?
> > > (BTW, I get the same error with == and !=)
> >
> > No! Not in general!
>
> Well,  this is a good point.  I think the current numerical behavior was
> a hack I stuck in for people who might not be aware of "is".  It's
> looking like a mistake now.
>
> > I learnt this back when Numeric implemented rich
> > comparisions; suddenly, lots of my code broke. You don't actually want
> > "is this object _equal_ (or not equal) to None", you want "is this object
> > None", as None is a singleton.
>
> However,  given the context of the original question, Sebastian's code
>
> doesn't read like *that* kind of None comparison:
> > > > > type '10a80' - that is, an array of 10 80 char 'strings' :
> > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field
> > > > > >>> q.Mrc.hdr('title') != None
>
> q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not
> None.  What did you want it to do Sebastian?

q.Mrc.hdr('title') is rather an array of 10 'character array' .. that's 
different, isn't it ?
In any case, I actually already changed my code to the check of 'is None' - 
which is probably what I wanted anyway. 
So, the only argument left, is that the original error/exception message I got 
was kind of "ugly" ;-) And I think/thought if 'something' cannot be 
_converted_ to a string array it can't _be_  '==' a string array. That's just 
why I expected the simple result to be 'False'.
But now I understand that if comparing an array to a scalar (maybe 'None') you 
always have to decide if this actually compares "the whole array" with the 
scalar, or if the comparison get "put through" and done on each element 
"separately" ...
which it seems is already decided at least in the numerical array case.


Regards,
Sebastian

Re: [Numpy-discussion] ObjectArray paramter order poll

[Paul B. <ba...@st...>]

Todd Miller wrote:
> A while back Colin Williams suggested that we change the parameter order
> of numarray.objects.ObjectArray from:
> 
> def __init__(self, shape=None, objects=None, rank=None)
> 
> to:
> 
> def __init__(self, objects=None, shape=None, rank=None)
> 
> The new order emphasizes building object arrays from existing sequences,
> while the old order emphasizes building empty (full of None) object
> arrays of a specific shape.  Since array() and fromlist() already exist
> to build from sequences,  I thought it was fine to keep the order as is,
> plus I hate breaking s/w interfaces unless it was my idea. :-)
> 
> Opinions please?  

+1  (for the second option)

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218

Re: [Numpy-discussion] comparing string array with None raises ValueError

[Todd M. <jm...@st...>]

On Thu, 2004-06-24 at 15:26, David M. Cooke wrote:
> On June 24, 2004 01:46 pm, Sebastian Haase wrote:
> 
> > In general it must be OK to compare anything with None, right ?
> > (BTW, I get the same error with == and !=)
> 
> No! Not in general! 

Well,  this is a good point.  I think the current numerical behavior was
a hack I stuck in for people who might not be aware of "is".  It's
looking like a mistake now.

> I learnt this back when Numeric implemented rich 
> comparisions; suddenly, lots of my code broke. You don't actually want "is 
> this object _equal_ (or not equal) to None", you want "is this object None", 
> as None is a singleton.

However,  given the context of the original question, Sebastian's code
doesn't read like *that* kind of None comparison:

> > > > type '10a80' - that is, an array of 10 80 char 'strings' :
> > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field
> > > > >>> q.Mrc.hdr('title') != None

q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not
None.  What did you want it to do Sebastian?

Regards,
Todd

RE: [Numpy-discussion] comparing string array with None raises ValueError, more opinions please

[Perry G. <pe...@st...>]

Todd Miller wrote:

> So the two choices now on the table are:
> 
> 1. Change string equality comparisons to return an array of false or
> true for objects which don't convert to strings.  Change Numeric
> equality comparisons in a similar fashion.
> 
> 2. Special case string equality comparisons for None, as is done with
> numerical comparisons.  Raise a type error for objects (other than None)
> which don't convert to string arrays.
> 
> I think I like 2.  Other opinions on 1 & 2?  Other choices?
> 
> Regards,
> Todd
> 
It would seem consistency with numerical arrays is important so 
unless someone presents a compelling argument for changing that
behavior for arrays, the choice appears clear. 

Perry

[wishing he had spent a little more time thinking about it and 
trying to remember why None is special cased for comparisons]

Re: [Numpy-discussion] comparing string array with None raises ValueError

[David M. C. <co...@ph...>]

On June 24, 2004 01:46 pm, Sebastian Haase wrote:

> In general it must be OK to compare anything with None, right ?
> (BTW, I get the same error with == and !=)

No! Not in general! I learnt this back when Numeric implemented rich 
comparisions; suddenly, lots of my code broke. You don't actually want "is 
this object _equal_ (or not equal) to None", you want "is this object None", 
as None is a singleton.

A contrived example:
>>> class A:
...     def __eq__(self, other): return True
...
>>> a = A()

So, this is bad:
>>> a == None
True

This is good:
>>> a is None
False

In general, if you're testing if something is None, use 'is', not '=='.
>>>q.Mrc.hdr('title') is not None

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

[Numpy-discussion] comparing string array with None raises ValueError, more opinions please

[Todd M. <jm...@st...>]

On Thu, 2004-06-24 at 14:53, Rick White wrote:
> On 24 Jun 2004, Todd Miller wrote:
> 
> > OK,  I see your point.  I talked it over with Perry and he made a
> > reasonable case for allowing comparisons with None (or any object).
> > Perry argued that since None is a common default parameter value, it
> > might simplify code to not have to add logic to handle that case.
> >
> > If no one objects,  I'll change numarray.strings so that comparison of a
> > string array with any object not convertible to a string array results
> > in an array of False values.
> >
> > Any objections?
> 
> Hmm, before you do that you might look at this:
> 
>     >>> import numarray
>     >>> b = numarray.zeros(10)
>     >>> b==0
>     array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool)
>     >>> b==None
>     0
> 
> So comparing numeric arrays to None returns a scalar false value since
> the variable b is not None.

Oh yeah...  I forgot about that.  Numeric comparisons have a special
case for None and blow up for most other arbitrary objects.  So there's
also:

>>> a= numarray.arange(100)
>>> a == None
0
>>> class foo:
..     pass
..
>>> a == foo
Traceback (most recent call last):
..
TypeError: UFunc arguments must be numarray, scalars or numeric
sequences

> I figure that the behavior of comparisons to None for string arrays
and
> numeric arrays ought to be consistent.  I don't know which is
> preferred...

So the two choices now on the table are:

1. Change string equality comparisons to return an array of false or
true for objects which don't convert to strings.  Change Numeric
equality comparisons in a similar fashion.

2. Special case string equality comparisons for None, as is done with
numerical comparisons.  Raise a type error for objects (other than None)
which don't convert to string arrays.

I think I like 2.  Other opinions on 1 & 2?  Other choices?

Regards,
Todd

Re: [Numpy-discussion] comparing string array with None raises ValueError

[Sebastian H. <ha...@ms...>]

On Thursday 24 June 2004 11:46 am, Todd Miller wrote:
> On Thu, 2004-06-24 at 13:46, Sebastian Haase wrote:
> > On Thursday 24 June 2004 10:31 am, Todd Miller wrote:
> > > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote:
> > > > Hi,
> > > > I'm not sure if this is fixed already in CVS but here it goes:
> > > > I'm working with record arrays, and trying to access a field with
> > > >
> > > > type '10a80' - that is, an array of 10 80 char 'strings' :
> > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field
> > > > >>> q.Mrc.hdr('title') != None
> > >
> > > Shouldn't this be:
> > >
> > > q.Mrc.hdr('title') != ""
> >
> > (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an
> > ARRAY of strings ! )
> >
> > No, I understand that this makes more sense, but I have some "display
> > hook"-code that compares everything with None...
> > In general it must be OK to compare anything with None, right ?
> > (BTW, I get the same error with == and !=)
>
> OK,  I see your point.  I talked it over with Perry and he made a
> reasonable case for allowing comparisons with None (or any object).
> Perry argued that since None is a common default parameter value, it
> might simplify code to not have to add logic to handle that case.
>
> If no one objects,  I'll change numarray.strings so that comparison of a
> string array with any object not convertible to a string array results
> in an array of False values.
>
> Any objections?

Just my feeling is, that a single None would do it ...

Regards,
Sebastian

Re: [Numpy-discussion] comparing string array with None raises ValueError

[Rick W. <rl...@st...>]

On 24 Jun 2004, Todd Miller wrote:

> OK,  I see your point.  I talked it over with Perry and he made a
> reasonable case for allowing comparisons with None (or any object).
> Perry argued that since None is a common default parameter value, it
> might simplify code to not have to add logic to handle that case.
>
> If no one objects,  I'll change numarray.strings so that comparison of a
> string array with any object not convertible to a string array results
> in an array of False values.
>
> Any objections?

Hmm, before you do that you might look at this:

    >>> import numarray
    >>> b = numarray.zeros(10)
    >>> b==0
    array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool)
    >>> b==None
    0

So comparing numeric arrays to None returns a scalar false value since
the variable b is not None.

I figure that the behavior of comparisons to None for string arrays and
numeric arrays ought to be consistent.  I don't know which is
preferred...
				Rick