numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] Comments on numarray - 4

[Edward C. J. <edc...@er...>]

The optional shape argument to fromstring is not documented.

-----------------

I suggest a documentation section on NumericTypes. "Int8.bytes" and
need documenting.

----------------

      Table of Automatic Coercions (found by experiment)

          B   I8   U8  I16  U16  I32  U32  I64  U64  F32  F64  C32  C64
      ------------------------------------------------------------------
    B |  *I8   I8   U8  I16  U16  I32  U32  I64  U64  F32  F64  C32  C64
   I8 |        I8 *I16  I16  U16  I32  U32  I64  U64  F32  F64  C32  C64
   U8 |             U8  I16  U16  I32  U32  I64  U64  F32  F64  C32  C64
  I16 |                 I16 *I32  I32  U32  I64  U64  F32  F64  C32  C64
  U16 |                      U16  I32  U32  I64  U64  F32  F64  C32  C64
  I32 |                           I32 *I64  I64  U64  F32  F64  C32  C64
  U32 |                                U32  I64  U64  F32  F64  C32  C64
  I64 |                                     I64 *I64 ?F32  F64  C32  C64
  U64 |                                          U64 ?F32  F64  C32  C64
  F32 |                                               F32  F64  C32  C64
  F64 |                                                    F64 ?C32  C64
  C32 |                                                         C32  C64
  C64 |                                                              C64

The names are listed in the order used in "numerictypes.py". The "*" 
cases are exceptions treated there. Are the "?" conversions correct? See 
"genericPromotionExclusions" in "numerictypes.py".

RE: [Numpy-discussion] Index array confusion...

[Perry G. <pe...@st...>]

> And performance, perhaps?
> 
Certainly. That was one big factor (more with regard to
memory demands than CPU demands).

> And; from Todd's comment about a hypothetical built-in for the
> purpose, I assume there is no way of copying the contents of one slice
> to another without going via a temporary array?
> 
If you mean:

Without overwriting the slice being copied to with overlapping slices
(whether it's a problem depends on how they overlap and whether one is
reversed or not, and one other effect described below) on the same array,
that's generally true.

In numarray, there are cases where the slice is copied internally in
blocks. It's possible that you could get away with

x[:] = x[-1::-1]

if x is smaller than the blocksize being used. You would be crazy
to depend on this though :-) For large enough arrays (try 10000 elements),
it definitely won't work.

Perry

[Numpy-discussion] Re: numarray questions

[Todd M. <jm...@st...>]

Mika Illouz wrote:

>Hi Todd,
>
>  
>
>>Thanks for the encouragement!   numarray-0.4 is coming out today...
>>    
>>
>
>Finally got around to playing with version 0.4 and the new NA_New (code
>attached).  Wnen I ran the test, I got the following trace:
>
>  
>
>>>>import _test_numarray
>>>>z = _test_numarray.t1()
>>>>        
>>>>
>Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File "/usr/lib/python2.2/site-packages/numarray/numarray.py", line 432, in __init__
>    bytestride=bytestride)
>  File "/usr/lib/python2.2/site-packages/numarray/ndarray.py", line 173, in __init__
>    self._data = memory.new_memory(size)
>OverflowError: long int too large to convert to int
>
>What am I doing wrong?
>
The parameter order in the call in t1() should be:

NA_New(a, tInt32, 1, length)

Because it is:

NA_New(a, tInt32, length, 1)

numarray thinks you're declaring a 5 dimensional array with some bizarre 
dimensions from "undefined values" on the stack.  One of these 
dimensions appears to be > MAX_INT.

>Thanks again,
>Mika
>  
>
Regards,
Todd

>  
>
>------------------------------------------------------------------------
>
>#include <Python.h>
>#include <libnumarray.h>
>
>static PyObject* t1( PyObject* self, PyObject* args )
>{
>  int length = 5;
>  int* a = new int[ length ];
>  for ( int i = 0 ; i < length ; i++ ) {
>    a[i] = i;
>  }
>
>  PyArrayObject* vec = NA_New( a, tInt32, length, 1 );
>  return (PyObject*)vec;
>  // return NA_ReturnOutput( NULL, vec );
>
>}
>
>static PyMethodDef TestNumarrayMethods[] = {
>  {"t1",  
>   t1, 
>   METH_VARARGS, 
>   "first test"
>  },
>  { NULL, NULL, 0, NULL}
>};
>
>extern "C" void init_test_numarray()
>{
>  Py_InitModule( "_test_numarray", TestNumarrayMethods );
>  import_libnumarray();
>}
>  
>

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>  
>
>And; from Todd's comment about a hypothetical built-in for the
>purpose, I assume there is no way of copying the contents of one slice
>to another without going via a temporary array?
>  
>
The problem there is with overlapping slices.

>Perry
>  
>

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Perry Greenfield <pe...@st...>:
>
> Magnus Lie Hetland wrote:
> 
> > >Well, what I was asking about was really what you meant by "correct".
> > >My interpretation of "correct" here was that only tuples and slices
> > >would be allowed as indexes.
> > >
> To clarify, tuples should be interpreted differently than lists or 
> arrays as arguments, as you suggested earlier so that
> x[1,2,3] is not interpreted the same as x[[1,2,3]] or x[array([1,2,3])].
> Granted, that will be confusing for those that try x[(1,2,3)]
> expecting it to be equivalent ot x[[1,2,3]] but we decided that
> the benefits of array indices well outweigh that confusing case.

Exactly. I agree completely with this (intended) behaviour. I was just
a bit baffled to find that numarray (0.4), in fact, behaved
differently. Since this is indeed a bug, the issue seems resolved to
me :)

(Except that I might have to add some version control to avoid running
with 0.4 :/)

> As far as far as slices resulting in views rather than copies
> (was with lists), this is a topic that has been talked to death.

Yes -- sorry about that.

> The original Numeric has view behavior.

I know. Just a mind-bug on my part :)

> When we started on numarray
> we were intending to make it more consistent with lists, but there
> were many that argued that view semantics were more sensible and
> we were persuaded that they were right (besides, the backward
> compatibility issue was very important as well).

And performance, perhaps?

And; from Todd's comment about a hypothetical built-in for the
purpose, I assume there is no way of copying the contents of one slice
to another without going via a temporary array?

> Perry

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

RE: [Numpy-discussion] Index array confusion...

[Perry G. <pe...@st...>]

Magnus Lie Hetland wrote:

> >Well, what I was asking about was really what you meant by "correct".
> >My interpretation of "correct" here was that only tuples and slices
> >would be allowed as indexes.
> >
To clarify, tuples should be interpreted differently than lists or 
arrays as arguments, as you suggested earlier so that
x[1,2,3] is not interpreted the same as x[[1,2,3]] or x[array([1,2,3])].
Granted, that will be confusing for those that try x[(1,2,3)]
expecting it to be equivalent ot x[[1,2,3]] but we decided that
the benefits of array indices well outweigh that confusing case.

As far as far as slices resulting in views rather than copies
(was with lists), this is a topic that has been talked to death.
The original Numeric has view behavior. When we started on numarray
we were intending to make it more consistent with lists, but there
were many that argued that view semantics were more sensible and
we were persuaded that they were right (besides, the backward
compatibility issue was very important as well).

Perry

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Magnus Lie Hetland <ma...@he...>:
>[snip]
>  
>
>>This seems like a bug to me. The assignment ought to swap the tails of
>>the sequences, as is the case with lists, but with numeric arrays,
>>some weird form of overwriting occurs. I guess this may be an
>>optimization (i.e. to avoid making copies), but it is rather
>>confusing. What do you think?
>>    
>>
>
>Even stranger:
>
>  
>
>>>>a = zeros([5])
>>>>b = ones([5])
>>>>p = 2
>>>>tail_a = a[p:]
>>>>tail_b = b[p:]
>>>>a[p:] = tail_b
>>>>b[p:] = tail_a
>>>>a, b
>>>>        
>>>>
>(array([0, 0, 1, 1, 1]), array([1, 1, 1, 1, 1]))
>
>I suppose this is due to sharing of slices somehow? I.e:
>  
>
This looks to me like the same problem,  just performing the effects of 
the tuple copy one step at a time.  The key is that the "tails" are 
views and not copies.

>  
>
>>>>a = ones([5])
>>>>b = a[:]
>>>>b[2] = 0
>>>>a
>>>>        
>>>>
>array([1, 1, 0, 1, 1])
>
>I have to use the copy method here, I guess?
>  
>
Yes.   array[ slice ] --> a view of a subarray.

>  
>
>>>Todd
>>>      
>>>
>
>  
>

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Todd Miller <jm...@st...>:
>  
>
>[snip]
>  
>
>>What I'm saying is that in the future it will work correctly.  :)
>>    
>>
>
>OK :)
>
>  
>
>>Most likely as you suggest,  but I haven't really looked at the code, 
>>just the rather sickening results.
>>    
>>
>
>Well, what I was asking about was really what you meant by "correct".
>My interpretation of "correct" here was that only tuples and slices
>would be allowed as indexes.
>
This actually worked as you expected in numarray-0.3.6.  The current 
behavior is a casualty of optimization to C.

>
>BTW: I found something else that I think is rather odd:
>
>  
>
>>>>x = [0, 0, 0]
>>>>y = [1, 1, 1]
>>>>p = 1
>>>>x[p:], y[p:] = y[p:], x[p:]
>>>>x, y
>>>>        
>>>>
>([0, 1, 1], [1, 0, 0])
>  
>
>>>>x = array([0, 0, 0])
>>>>y = array([1, 1, 1])
>>>>x[p:], y[p:] = y[p:], x[p:]
>>>>x, y
>>>>        
>>>>
>(array([0, 1, 1]), array([1, 1, 1]))
>
>This seems like a bug to me. The assignment ought to swap the tails of
>
Numeric does this as well,  so I would not call it a bug,  but  I agree 
that it is unfortunate.

>the sequences, as is the case with lists, but with numeric arrays,
>some weird form of overwriting occurs. I guess this may be an
>optimization (i.e. to avoid making copies), but it is rather
>
I think you're correct here.   This behavior is a consequence of the 
fact that array slices are views and not copies.  To fix it,  just say:

x[p:], y[p:] = y[p:].copy(), x[p:].copy()

>confusing. What do you think?
>
The manual should probably document this as a gotcha, and we might want 
to consider adding an exchange ufunc which can do the swap without a 
temporary.   I doubt the cost of exchange is worth it though.

Thanks for the feedback,
Todd

[Numpy-discussion] "not"

[Magnus L. H. <ma...@he...>]

Just a quick note: Why does the operator 'not' appear in parentheses
beside logical_not in the manual? Isn't that a bit misleading (since
it doesn't work the same way, as opposed to 'and' and 'or')?

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Magnus Lie Hetland <ma...@he...>:
[snip]
> This seems like a bug to me. The assignment ought to swap the tails of
> the sequences, as is the case with lists, but with numeric arrays,
> some weird form of overwriting occurs. I guess this may be an
> optimization (i.e. to avoid making copies), but it is rather
> confusing. What do you think?

Even stranger:

>>> a = zeros([5])
>>> b = ones([5])
>>> p = 2
>>> tail_a = a[p:]
>>> tail_b = b[p:]
>>> a[p:] = tail_b
>>> b[p:] = tail_a
>>> a, b
(array([0, 0, 1, 1, 1]), array([1, 1, 1, 1, 1]))

I suppose this is due to sharing of slices somehow? I.e:

>>> a = ones([5])
>>> b = a[:]
>>> b[2] = 0
>>> a
array([1, 1, 0, 1, 1])

I have to use the copy method here, I guess?

> > Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
>
[snip]
> What I'm saying is that in the future it will work correctly.  :)

OK :)

> Most likely as you suggest,  but I haven't really looked at the code, 
> just the rather sickening results.

Well, what I was asking about was really what you meant by "correct".
My interpretation of "correct" here was that only tuples and slices
would be allowed as indexes.

BTW: I found something else that I think is rather odd:

>>> x = [0, 0, 0]
>>> y = [1, 1, 1]
>>> p = 1
>>> x[p:], y[p:] = y[p:], x[p:]
>>> x, y
([0, 1, 1], [1, 0, 0])
>>> x = array([0, 0, 0])
>>> y = array([1, 1, 1])
>>> x[p:], y[p:] = y[p:], x[p:]
>>> x, y
(array([0, 1, 1]), array([1, 1, 1]))

This seems like a bug to me. The assignment ought to swap the tails of
the sequences, as is the case with lists, but with numeric arrays,
some weird form of overwriting occurs. I guess this may be an
optimization (i.e. to avoid making copies), but it is rather
confusing. What do you think?

> Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
[snip]
> Assuming you're talking about numarray here,

Indeed.

> I regard this as a 0.4 bug which you just found.  Thanks!

No problem :)

So, I guess you're saying that in the future you will typecheck for
tuples and slice objects vs. other objects (such as arrays and lists)?

> Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>I can see why this happens, to some degree, but it is a bit confusing
>still:
>
>  
>
>>>>a = zeros([5,5])
>>>>a[[1,2,3]]
>>>>        
>>>>
>array([[0, 0, 0, 0, 0],
>       [0, 0, 0, 0, 0],
>       [0, 0, 0, 0, 0]])
>  
>
>>>>a[[1,2]]
>>>>        
>>>>
>0
>  
>
>>>>a[[1]]
>>>>        
>>>>
>array([[0, 0, 0, 0, 0]])
>
>I was planning to use index arrays to select a subset of the rows in a
>two-dimensional array, but because of the special-casing for
>one-dimensional arrays of the same length as the dimensionality of the
>array being indexed, it seems I can't do that (without using
>put/take, which may be quite OK, I guess)... Am I right? (I see why
>this behaviour is needed for tuples and slice objects; and I guess
>type checking would be sort of "ugly"...)
>
>Of course a[[1,2],] works just as expected.
>
>This may not be important, but perhaps worth a sentence or example in
>the manual? I.e. if you are using an index array idx and you don't
>want to risk having it cast as a single index lookup, you should do
>
>  a[idx,]
>
>Wouldn't that be sound advice in general?
>
>  
>
Assuming you're talking about numarray here,  I regard this as a 0.4 bug 
which you just found.  Thanks!

Todd

[Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

I can see why this happens, to some degree, but it is a bit confusing
still:

>>> a = zeros([5,5])
>>> a[[1,2,3]]
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])
>>> a[[1,2]]
0
>>> a[[1]]
array([[0, 0, 0, 0, 0]])

I was planning to use index arrays to select a subset of the rows in a
two-dimensional array, but because of the special-casing for
one-dimensional arrays of the same length as the dimensionality of the
array being indexed, it seems I can't do that (without using
put/take, which may be quite OK, I guess)... Am I right? (I see why
this behaviour is needed for tuples and slice objects; and I guess
type checking would be sort of "ugly"...)

Of course a[[1,2],] works just as expected.

This may not be important, but perhaps worth a sentence or example in
the manual? I.e. if you are using an index array idx and you don't
want to risk having it cast as a single index lookup, you should do

  a[idx,]

Wouldn't that be sound advice in general?

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

[Numpy-discussion] Comments on numarray - 3

[Edward C. J. <edc...@er...>]

NA_OFFSETDATA is mentioned during the example in 10.2.3. It should be 
documented in 10.2.1. Should some of the other macros in "nummacro.h" be 
documented? Also the macros in "arrayobject.h" and "numcomplex.h".

-------------------

The "libnumarray_UNIQUE_SYMBOL" trick discussed in "libnumarray.h" has 
been useful with Numeric and will be useful with numarray. It needs to 
be in the main documentation. I think it is mentioned somewhere in the 
Python documentation. I found it very confusing the first time I used 
it. A detailed example is needed.

-------------------

"NA_longsFromIntTuple" works on both tuples and lists. A better name 
might be "NA_longsFromIntSequence". Functions of this type have proven 
_very_ useful for me in writing large SWIG wrappers. I generate them 
using SWIG macros. Here are some of them. On the C side, auxillary 
information about the lengths of things needs to be dealt with. "list" 
means "list or tuple".

Python                     C
list of ints          <--> array of ints
list of floats        <--> array of doubles
list of floats        <--> array of floats
...
list of strings       <--> array of arrays of char
list of lists of ints <--> array of arrays of int
...
list                  <--> array of PyObject*

Thanks,
Ed Jones

[Numpy-discussion] Comments on Documentation - 2

[Edward C. J. <edc...@er...>]

I am currently learning about numarray. Here are further comments on the 
documentation.

----------------------

"libnumarray.h" and "numarray.h" are never mentioned in the docs.

----------------------

10.2.3 Is "free(result)" needed here?

         double *result;
         int m, n;
         .
         .
         .
         result = func(...);
         if(NULL == result)
             return NULL;
         return NA_New((void *)result, tFloat64, m, n, 2);

----------------------

10.6 "NA_New", "NA_Empty", and "NA_NewA11" return a "PyArrayObject*", 
not a "PyObject*".

RE: [Numpy-discussion] 3 comments on numarray documentation

[Perry G. <pe...@st...>]

Sorry it took so long to respond. We appreciate this feedback.

Edward C. Jones writes:
 
> To access UInt8, etc,
> 
>      from numerictypes import *
> 
> Maybe mention this in 4.2.1.
> 
Are you referring to text in another part of the manual or are you
suggesting that this be added to 4.2.1? If added I would reword it
somewhat since these type names are in the numarray namespace. If 
one wants to do:

import numarray

and have the types as part of you namespace it would make sense to

import numarray
from numerictypes import *

(though if we go to a package system, this may become
from numarray.numerictypes import *)

We also need to add the fact that there are now 
UInt32, UInt64 (not on windows), Int64 types.

> -------
> 
> In 4.7
> 
>       Only one "..." is expanded in an index expression, so if one has a
>       rank-5 array C, then C[...,0,...] is the same thing as
>       C[:,:,:,0,:].
> 
> So an unexpanded "..." is treated as a ':'?
> 
yes

> ----------
> 
> In 5.1.1,
> 
>       >>> a = arange(5, type=Float64)
>       >>> print a[::-1] * 1.2
>       [ 4.8  3.6  2.4  1.2  0. ]
>       >>> multiply(a[::-1], 1.2, a)
>       >>> a
>       array([ 4.8 ,  3.6 ,  2.4 ,  1.2,  0. ])
> 
> doesn't make the desired point. Try:
> 
>       >>> a = arange(5, type=Int32)
>       >>> a
>       [0 1 2 3 4]
>       >>> print a[::-1] * 1.2
>       [ 4.8  3.6  2.4  1.2  0. ]
>       >>> multiply(a[::-1], 1.2, a)
>       >>> a
>       array([4 3 2 1 0])
> 
Yes, we will make this change

> Why does Python documentation always use the interpreter? I doubt if it 
> is used much. Why not:
> 
>       a = arange(5, type=Int32)
>       print a.type(), a
>       b = a[::-1] * 1.2
>       print b.type(), b
>       numarray.multiply(a[::-1], 1.2, a)
>       print a.type(), a
> 
Actually many do use it in interpreter mode, at least here. But I think
you miss the main point which is to show the result of each command
for the purposes of instruction. Even if you never plan to use the
interpreter (which I think would be a mistake since it is a wonderful
way of verifying that things work the way you thought they did),
it serves to show examples in a very clear and concise way. 

Perry

Re: [Numpy-discussion] Some questions about PyArray_As2D.

[Konrad H. <hi...@cn...>]

> The function prototype says:
> 
> extern int PyArray_As2D(PyObject **op, char ***ptr, int *d1, int *d2, 
> int typecode)

You are right that my first comment doesn't quite apply, I hadn't
noticed the two stars...

But the code is still OK, it is the calling routine that is responsible
for avoiding memory leaks.

> is exactly what you do with the instruction:
> 
> *op = (PyObject *)ap;
> 
> So you create a new PyArrayObject (allocating another memory area) by 
> means of PyArray_ContiguousFromObject and names it ap, then you modify 
> the memory address which op points to with the above instruction. Now 

Right. This routine cannot know if that reference contains an "owned"
or a "borrowed" reference. In the first case, the reference counter
of the original array must be decreased, in the second case not.

Assume for example that the calling routine got an array passed in
as a borrowed reference:

void foo(PyObject *array)
{
   /* I want a 2D version! */
   PyArray_As2D(&array, ...)
}

In that case, decreasing the reference count in PyArray_As2D would
be disastrous.

In the case of an owned reference, the calling routine must keep a
copy of the pointer, call PyArray_As2D, and then decrease the
reference counter on the original pointer. This ought to be easier.
In fact, the interface of PyArray_As2D is pretty badly designed.
It should not overwrite the original pointer, but return a new
one. I also don't see the point in passing all the array information
in additional arguments - they are easy to obtain from the new
array object.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] Some questions about PyArray_As2D.

[Andrea R. <ari...@pi...>]

On Wednesday, Dec 11, 2002, at 19:43 Europe/Rome, Konrad Hinsen wrote:

> No. op is an input parameter and thus a "borrowed" reference. It might
> not be the best coding style to reuse that variable name for something
> unrelated later on, but it doesn't cause a memory leak.

I'm sorry but I don't agree. I've read your answer many times and 
re-read the code but I still think to be right.

The function prototype says:

extern int PyArray_As2D(PyObject **op, char ***ptr, int *d1, int *d2, 
int typecode)

and a call to this function looks like this:

PyObject *input;
double **result;
int nrows, ncols;

PyArray_As2D((&input, (char ***) &(result), &(nrows), &(ncols), 
PyArray_DOUBLE)

Now when you call the function in this way op is a pointer to the 
pointer that points to your original ArrayObject. It allows you to 
change the memory address which is originally pointed by input. And it 
is exactly what you do with the instruction:

*op = (PyObject *)ap;

So you create a new PyArrayObject (allocating another memory area) by 
means of PyArray_ContiguousFromObject and names it ap, then you modify 
the memory address which op points to with the above instruction. Now 
*op (that is input) points to another memory region, but you haven't 
deallocated the previous pointed memory and it _is_ a memory leak!

These are my two cents, comments are welcome.

Cheers,
Andrea.


---
Andrea Riciputi        <mailto:and...@li...>

"Science is like sex: sometimes something useful comes out,
   but that is not the reason we are doing it" -- (Richard Feynman)

[Numpy-discussion] real/imag for non complex numbers [was numarray complex comparisons]

[Travis O. <oli...@ie...>]

In investigating, why Numeric does not allow a.real for non-complex
arrays which many have argued that it should, I think I uncovered an old
bug in Numeric.

In looking at the code, it appears that the intention was to have a.real
just return a new array (pointing to the same data as a but having a new
header) whenever a was not a complex array.  Unfortunately, the code had
a bug and did not actually return the newly created array.  As a result,
the default behavior of raising an "Attribute not found" exception
occurred.  But, before this happened, the array the reference count on
the array occurred, thereby springing a memory leak. 

Try this out.

a = ones((10,10))

print sys.getrefcount(a)

a.real   # will generate an exception

print sys.getrefcount(a)

a.real  

print sys.getrefcount(a)

Notice that each time a.real is called the reference count for a goes up
by one even though no object is returned.


I have fixed the bug in Numeric CVS by returning the created object for
a.real.

The end result is that a.real works for all arrays (not just
complex-valued ones). 

-Travis O.



On Wed, 2002-12-11 at 16:24, Tim Hochberg wrote:
> 
> Here's my latest thinking on this. I'll put this in the form of a proposal
> so that it's easier to comment on. Please comment on part I and part II
> separately.
> 
> I. Attributes real and imag/imaginary[1]
> 
> All numeric array types (i.e., excluding character and object arrays) would
> have both a real and imag/imaginary attribute. For complex types ('F', 'D')
> this would behave just as it does now. For non-complex types ('i', 'f', 'd',
> etc.), the real attribute would return the object itself, while the
> imaginary attribute would return a ReadOnlyZeroArray [2] of the correct
> shape and typecode. I think that this would provide behaviour consistent
> with the current real and imag attributes on complex arrays without
> introducing and spooky corner cases (that statement is asking for trouble!).
> Examples:
> 
> >>> a = array([1,2,3])
> >>> a.real[0] = 99
> >>> a.real
> array([99, 2, 3])
> >>> a.imag
> array([0,0,0])
> >>> a.imag[0] = 99
> TypeError: object is read only.
> 
> II. Functions real and imag
> 
> Two functions real and imag (or imaginary or both) would be provided. The
> would behave as if they were defined as shown below, even if item I. is not
> adopted.
> 
> def real(a):
>     return asarray(a).real
> 
> def imag(a):
>     return asarray(a).imag
> 
> 
> Comments?
> 
> I'm currently +0 on I. and +1 on II.
> 
> -tim
> 
> 
> 
> [1] Numeric arrays haveboth imag and imaginary attributes, but they are
> identical. I'm not sure what Numarray's plans are in this regard.
> 
> [2] A ReadOnlyZeroArray(shape, typecode) would behave like an array of zeros
> with the given shape and typecode for all non mutating operations. For
> mutation operations (a[x] = b, a += c, etc.) it would raise an exception.
> This object does not actually allocate any space, so creation is fast and
> memory efficient. Implementing this might be the hardest part of the
> proposal.
> 
> 
> _______________________________________________
> SciPy-user mailing list
> Sci...@sc...
> http://www.scipy.net/mailman/listinfo/scipy-user

Re: [Numpy-discussion] 3 comments on numarray documentation

[Chris B. <Chr...@no...>]

"Edward C. Jones" wrote:

> Why does Python documentation always use the interpreter? I doubt if it
> is used much. 

I, for one, use it all the time to explore the syntax and function of a
simple statement, just like the examples.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

[Numpy-discussion] 3 comments on numarray documentation

[Edward C. J. <edc...@er...>]

To access UInt8, etc,

     from numerictypes import *

Maybe mention this in 4.2.1.

-------

In 4.7

      Only one "..." is expanded in an index expression, so if one has a
      rank-5 array C, then C[...,0,...] is the same thing as
      C[:,:,:,0,:].

So an unexpanded "..." is treated as a ':'?

----------

In 5.1.1,

      >>> a = arange(5, type=Float64)
      >>> print a[::-1] * 1.2
      [ 4.8  3.6  2.4  1.2  0. ]
      >>> multiply(a[::-1], 1.2, a)
      >>> a
      array([ 4.8 ,  3.6 ,  2.4 ,  1.2,  0. ])

doesn't make the desired point. Try:

      >>> a = arange(5, type=Int32)
      >>> a
      [0 1 2 3 4]
      >>> print a[::-1] * 1.2
      [ 4.8  3.6  2.4  1.2  0. ]
      >>> multiply(a[::-1], 1.2, a)
      >>> a
      array([4 3 2 1 0])

Why does Python documentation always use the interpreter? I doubt if it 
is used much. Why not:

      a = arange(5, type=Int32)
      print a.type(), a
      b = a[::-1] * 1.2
      print b.type(), b
      numarray.multiply(a[::-1], 1.2, a)
      print a.type(), a

Re: [Numpy-discussion] Some questions about PyArray_As2D.

[Konrad H. <hi...@cn...>]

> Just done. By the way reading the code again and again I got another 
> question. Here is the complete code fragment:
> 
> extern int PyArray_As2D(PyObject **op, char ***ptr, int *d1, int *d2, 
> int typecode) {
>      PyArrayObject *ap;
>      int i, n;
>      char **data;
> 
>      if ((ap = (PyArrayObject *)PyArray_ContiguousFromObject(*op, 
> typecode, 2, 2)) == NULL)
>          return -1;
> 
>      n = ap->dimensions[0];
>      data = (char **)malloc(n*sizeof(char *));
>      for(i=0; i<n; i++) {
>          data[i] = ap->data + i*ap->strides[0];
>      }
>      *op = (PyObject *)ap;  <=== It doesn't sound good to me!!!
>      *ptr = data;
>      *d1 = ap->dimensions[0];
>      *d2 = ap->dimensions[1];
>      return 0;
> }
> 
> Looking at the marked line I started wondering about the fate of the 
> object originally pointed by op. Without explicitly deallocating it you 
> lost any chance to reach it. It turns out in a memory leakage. 

No. op is an input parameter and thus a "borrowed" reference. It might
not be the best coding style to reuse that variable name for something
unrelated later on, but it doesn't cause a memory leak.

> I'm very interested in this topic because I'm writing some Python 
> extensions and I'd like to understand how I have to handle all these 
> objects correctly. So how "long" does a Python object live? How can I 
> release correctly the allocated memory?

There is an explanation of this topic in chapter 1.10 of the Python
"Extending and Embedding" manual. Basically, you have to increase an
object's reference counter if you want to keep a reference to it
beyond the end of the currently running function, and to decrease the
counter if you want to release that reference. Whenever the reference
count goes to zero, the object is deleted. With very few exceptions,
functions that take an object as input to work on (but not store)
don't increase the reference counter. They don't have to, because
nothing can happen to the object until the function terminates.
Special precautions need to be taken for multithreading, but these go
much beyond object reference counting.

As a rule of thumb, you don't have to worry about reference counting
at all as long as you only write C functions that act on existing
objects. It becomes an issue when you define your own extension types
in C.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] Some questions about PyArray_As2D.

[Andrea R. <ari...@pi...>]

On Wednesday, Dec 11, 2002, at 14:33 Europe/Rome, Konrad Hinsen wrote:

>> 2) Next in the source code I've found the following memory allocation:
>>
>> data = (char **)malloc(n*sizeof(char *));
>>
>> without checking if malloc return NULL or not. As far as I know it's
>> not safe, even if it's very unlikely that this malloc would fail.
>
> Right. Did you submit a bug report?

Just done. By the way reading the code again and again I got another 
question. Here is the complete code fragment:

extern int PyArray_As2D(PyObject **op, char ***ptr, int *d1, int *d2, 
int typecode) {
     PyArrayObject *ap;
     int i, n;
     char **data;

     if ((ap = (PyArrayObject *)PyArray_ContiguousFromObject(*op, 
typecode, 2, 2)) == NULL)
         return -1;

     n = ap->dimensions[0];
     data = (char **)malloc(n*sizeof(char *));
     for(i=0; i<n; i++) {
         data[i] = ap->data + i*ap->strides[0];
     }
     *op = (PyObject *)ap;  <=== It doesn't sound good to me!!!
     *ptr = data;
     *d1 = ap->dimensions[0];
     *d2 = ap->dimensions[1];
     return 0;
}

Looking at the marked line I started wondering about the fate of the 
object originally pointed by op. Without explicitly deallocating it you 
lost any chance to reach it. It turns out in a memory leakage. 
Obviously the same problem happend in PyArray_As1D function.

I'm very interested in this topic because I'm writing some Python 
extensions and I'd like to understand how I have to handle all these 
objects correctly. So how "long" does a Python object live? How can I 
release correctly the allocated memory?

Thanks,
Andrea.
---
Andrea Riciputi        <mailto:and...@li...>

"Science is like sex: sometimes something useful comes out,
   but that is not the reason we are doing it" -- (Richard Feynman)

Re: [Numpy-discussion] Some questions about PyArray_As2D.

[Konrad H. <hi...@cn...>]

Andrea Riciputi <ari...@pi...> writes:

> As far as I can understand the source version is correct while the
> manual's one is not. Am I wrong?

The source code always wins the argument :-)

> 2) Next in the source code I've found the following memory allocation:
> 
> data = (char **)malloc(n*sizeof(char *));
> 
> without checking if malloc return NULL or not. As far as I know it's
> not safe, even if it's very unlikely that this malloc would fail.

Right. Did you submit a bug report?

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------