numpy-discussion

[Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

I can see why this happens, to some degree, but it is a bit confusing
still:

>>> a = zeros([5,5])
>>> a[[1,2,3]]
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])
>>> a[[1,2]]
0
>>> a[[1]]
array([[0, 0, 0, 0, 0]])

I was planning to use index arrays to select a subset of the rows in a
two-dimensional array, but because of the special-casing for
one-dimensional arrays of the same length as the dimensionality of the
array being indexed, it seems I can't do that (without using
put/take, which may be quite OK, I guess)... Am I right? (I see why
this behaviour is needed for tuples and slice objects; and I guess
type checking would be sort of "ugly"...)

Of course a[[1,2],] works just as expected.

This may not be important, but perhaps worth a sentence or example in
the manual? I.e. if you are using an index array idx and you don't
want to risk having it cast as a single index lookup, you should do

  a[idx,]

Wouldn't that be sound advice in general?

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>I can see why this happens, to some degree, but it is a bit confusing
>still:
>
>  
>
>>>>a = zeros([5,5])
>>>>a[[1,2,3]]
>>>>        
>>>>
>array([[0, 0, 0, 0, 0],
>       [0, 0, 0, 0, 0],
>       [0, 0, 0, 0, 0]])
>  
>
>>>>a[[1,2]]
>>>>        
>>>>
>0
>  
>
>>>>a[[1]]
>>>>        
>>>>
>array([[0, 0, 0, 0, 0]])
>
>I was planning to use index arrays to select a subset of the rows in a
>two-dimensional array, but because of the special-casing for
>one-dimensional arrays of the same length as the dimensionality of the
>array being indexed, it seems I can't do that (without using
>put/take, which may be quite OK, I guess)... Am I right? (I see why
>this behaviour is needed for tuples and slice objects; and I guess
>type checking would be sort of "ugly"...)
>
>Of course a[[1,2],] works just as expected.
>
>This may not be important, but perhaps worth a sentence or example in
>the manual? I.e. if you are using an index array idx and you don't
>want to risk having it cast as a single index lookup, you should do
>
>  a[idx,]
>
>Wouldn't that be sound advice in general?
>
>  
>
Assuming you're talking about numarray here,  I regard this as a 0.4 bug 
which you just found.  Thanks!

Todd

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
[snip]
> Assuming you're talking about numarray here,

Indeed.

> I regard this as a 0.4 bug which you just found.  Thanks!

No problem :)

So, I guess you're saying that in the future you will typecheck for
tuples and slice objects vs. other objects (such as arrays and lists)?

> Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
>
[snip]
> What I'm saying is that in the future it will work correctly.  :)

OK :)

> Most likely as you suggest,  but I haven't really looked at the code, 
> just the rather sickening results.

Well, what I was asking about was really what you meant by "correct".
My interpretation of "correct" here was that only tuples and slices
would be allowed as indexes.

BTW: I found something else that I think is rather odd:

>>> x = [0, 0, 0]
>>> y = [1, 1, 1]
>>> p = 1
>>> x[p:], y[p:] = y[p:], x[p:]
>>> x, y
([0, 1, 1], [1, 0, 0])
>>> x = array([0, 0, 0])
>>> y = array([1, 1, 1])
>>> x[p:], y[p:] = y[p:], x[p:]
>>> x, y
(array([0, 1, 1]), array([1, 1, 1]))

This seems like a bug to me. The assignment ought to swap the tails of
the sequences, as is the case with lists, but with numeric arrays,
some weird form of overwriting occurs. I guess this may be an
optimization (i.e. to avoid making copies), but it is rather
confusing. What do you think?

> Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Magnus Lie Hetland <ma...@he...>:
[snip]
> This seems like a bug to me. The assignment ought to swap the tails of
> the sequences, as is the case with lists, but with numeric arrays,
> some weird form of overwriting occurs. I guess this may be an
> optimization (i.e. to avoid making copies), but it is rather
> confusing. What do you think?

Even stranger:

>>> a = zeros([5])
>>> b = ones([5])
>>> p = 2
>>> tail_a = a[p:]
>>> tail_b = b[p:]
>>> a[p:] = tail_b
>>> b[p:] = tail_a
>>> a, b
(array([0, 0, 1, 1, 1]), array([1, 1, 1, 1, 1]))

I suppose this is due to sharing of slices somehow? I.e:

>>> a = ones([5])
>>> b = a[:]
>>> b[2] = 0
>>> a
array([1, 1, 0, 1, 1])

I have to use the copy method here, I guess?

> > Todd

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Magnus Lie Hetland <ma...@he...>:
>[snip]
>  
>
>>This seems like a bug to me. The assignment ought to swap the tails of
>>the sequences, as is the case with lists, but with numeric arrays,
>>some weird form of overwriting occurs. I guess this may be an
>>optimization (i.e. to avoid making copies), but it is rather
>>confusing. What do you think?
>>    
>>
>
>Even stranger:
>
>  
>
>>>>a = zeros([5])
>>>>b = ones([5])
>>>>p = 2
>>>>tail_a = a[p:]
>>>>tail_b = b[p:]
>>>>a[p:] = tail_b
>>>>b[p:] = tail_a
>>>>a, b
>>>>        
>>>>
>(array([0, 0, 1, 1, 1]), array([1, 1, 1, 1, 1]))
>
>I suppose this is due to sharing of slices somehow? I.e:
>  
>
This looks to me like the same problem,  just performing the effects of 
the tuple copy one step at a time.  The key is that the "tails" are 
views and not copies.

>  
>
>>>>a = ones([5])
>>>>b = a[:]
>>>>b[2] = 0
>>>>a
>>>>        
>>>>
>array([1, 1, 0, 1, 1])
>
>I have to use the copy method here, I guess?
>  
>
Yes.   array[ slice ] --> a view of a subarray.

>  
>
>>>Todd
>>>      
>>>
>
>  
>

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Todd Miller <jm...@st...>:
>  
>
>[snip]
>  
>
>>What I'm saying is that in the future it will work correctly.  :)
>>    
>>
>
>OK :)
>
>  
>
>>Most likely as you suggest,  but I haven't really looked at the code, 
>>just the rather sickening results.
>>    
>>
>
>Well, what I was asking about was really what you meant by "correct".
>My interpretation of "correct" here was that only tuples and slices
>would be allowed as indexes.
>
This actually worked as you expected in numarray-0.3.6.  The current 
behavior is a casualty of optimization to C.

>
>BTW: I found something else that I think is rather odd:
>
>  
>
>>>>x = [0, 0, 0]
>>>>y = [1, 1, 1]
>>>>p = 1
>>>>x[p:], y[p:] = y[p:], x[p:]
>>>>x, y
>>>>        
>>>>
>([0, 1, 1], [1, 0, 0])
>  
>
>>>>x = array([0, 0, 0])
>>>>y = array([1, 1, 1])
>>>>x[p:], y[p:] = y[p:], x[p:]
>>>>x, y
>>>>        
>>>>
>(array([0, 1, 1]), array([1, 1, 1]))
>
>This seems like a bug to me. The assignment ought to swap the tails of
>
Numeric does this as well,  so I would not call it a bug,  but  I agree 
that it is unfortunate.

>the sequences, as is the case with lists, but with numeric arrays,
>some weird form of overwriting occurs. I guess this may be an
>optimization (i.e. to avoid making copies), but it is rather
>
I think you're correct here.   This behavior is a consequence of the 
fact that array slices are views and not copies.  To fix it,  just say:

x[p:], y[p:] = y[p:].copy(), x[p:].copy()

>confusing. What do you think?
>
The manual should probably document this as a gotcha, and we might want 
to consider adding an exchange ufunc which can do the swap without a 
temporary.   I doubt the cost of exchange is worth it though.

Thanks for the feedback,
Todd

RE: [Numpy-discussion] Index array confusion...

[Perry G. <pe...@st...>]

Magnus Lie Hetland wrote:

> >Well, what I was asking about was really what you meant by "correct".
> >My interpretation of "correct" here was that only tuples and slices
> >would be allowed as indexes.
> >
To clarify, tuples should be interpreted differently than lists or 
arrays as arguments, as you suggested earlier so that
x[1,2,3] is not interpreted the same as x[[1,2,3]] or x[array([1,2,3])].
Granted, that will be confusing for those that try x[(1,2,3)]
expecting it to be equivalent ot x[[1,2,3]] but we decided that
the benefits of array indices well outweigh that confusing case.

As far as far as slices resulting in views rather than copies
(was with lists), this is a topic that has been talked to death.
The original Numeric has view behavior. When we started on numarray
we were intending to make it more consistent with lists, but there
were many that argued that view semantics were more sensible and
we were persuaded that they were right (besides, the backward
compatibility issue was very important as well).

Perry

Re: [Numpy-discussion] Index array confusion...

[Magnus L. H. <ma...@he...>]

Perry Greenfield <pe...@st...>:
>
> Magnus Lie Hetland wrote:
> 
> > >Well, what I was asking about was really what you meant by "correct".
> > >My interpretation of "correct" here was that only tuples and slices
> > >would be allowed as indexes.
> > >
> To clarify, tuples should be interpreted differently than lists or 
> arrays as arguments, as you suggested earlier so that
> x[1,2,3] is not interpreted the same as x[[1,2,3]] or x[array([1,2,3])].
> Granted, that will be confusing for those that try x[(1,2,3)]
> expecting it to be equivalent ot x[[1,2,3]] but we decided that
> the benefits of array indices well outweigh that confusing case.

Exactly. I agree completely with this (intended) behaviour. I was just
a bit baffled to find that numarray (0.4), in fact, behaved
differently. Since this is indeed a bug, the issue seems resolved to
me :)

(Except that I might have to add some version control to avoid running
with 0.4 :/)

> As far as far as slices resulting in views rather than copies
> (was with lists), this is a topic that has been talked to death.

Yes -- sorry about that.

> The original Numeric has view behavior.

I know. Just a mind-bug on my part :)

> When we started on numarray
> we were intending to make it more consistent with lists, but there
> were many that argued that view semantics were more sensible and
> we were persuaded that they were right (besides, the backward
> compatibility issue was very important as well).

And performance, perhaps?

And; from Todd's comment about a hypothetical built-in for the
purpose, I assume there is no way of copying the contents of one slice
to another without going via a temporary array?

> Perry

-- 
Magnus Lie Hetland        Practical Python          The Anygui Project
http://hetland.org        http://ppython.com        http://anygui.org

Re: [Numpy-discussion] Index array confusion...

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>  
>
>And; from Todd's comment about a hypothetical built-in for the
>purpose, I assume there is no way of copying the contents of one slice
>to another without going via a temporary array?
>  
>
The problem there is with overlapping slices.

>Perry
>  
>

RE: [Numpy-discussion] Index array confusion...

[Perry G. <pe...@st...>]

> And performance, perhaps?
> 
Certainly. That was one big factor (more with regard to
memory demands than CPU demands).

> And; from Todd's comment about a hypothetical built-in for the
> purpose, I assume there is no way of copying the contents of one slice
> to another without going via a temporary array?
> 
If you mean:

Without overwriting the slice being copied to with overlapping slices
(whether it's a problem depends on how they overlap and whether one is
reversed or not, and one other effect described below) on the same array,
that's generally true.

In numarray, there are cases where the slice is copied internally in
blocks. It's possible that you could get away with

x[:] = x[-1::-1]

if x is smaller than the blocksize being used. You would be crazy
to depend on this though :-) For large enough arrays (try 10000 elements),
it definitely won't work.

Perry