numpy-discussion

[Numpy-discussion] take for character arrays?

[Faheem M. <fa...@em...>]

Hi,

I'm trying to extract subarrays of a character array using take(), but
getting mysterious errors, for example:

ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

I am thinking that perhaps this operation is not supported for
character arrays, and I should convert my character array to a numeric
array instead (the strings are all of length 1).

Advice, suggestions, enlightenment appreciated.

                                                          Faheem.

[Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004 05:13:29 +0000 (UTC), Faheem Mitha
<fa...@em...> wrote:

> Hi,
>
> I'm trying to extract subarrays of a character array using take(), but
> getting mysterious errors, for example:
>
> ValueError: Invalid destination array: partial indices require
> contiguous non-byteswapped destination
>
> I am thinking that perhaps this operation is not supported for
> character arrays, and I should convert my character array to a numeric
> array instead (the strings are all of length 1).

I read this again and realised this does not give any specific
information, and is not very coherent. Got to avoid those late night
postings. So here are the details. If I set axis = 0 there is no
problem. I also don't have this problem with numerical arrays.

I am sure there is a simple reason for this, and hopefully a simple
workaround.

Thanks in advance.

                                                          Faheem.

**************************************************************************
In [22]: import numarray.strings as str

In [23]: import numarray

In [24]: s = str.array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h',
'i'],shape=(3,3))

In [25]: numarray.take(s,(1,2),axis=1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call
last)

/home/faheem/wc/corrmodel/boost/<console>

/usr/lib/python2.3/site-packages/numarray/ufunc.py in take(array,
indices, axis, outarr, clipmode)
   1779             raise ValueError("indices must be a sequence")
   1780         work = _gen.swapaxes(array, 0, axis)
-> 1781         work = work._take((indices,), outarr=outarr,
clipmode=clipmode)
   1782         return _gen.swapaxes(work, 0, axis)
   1783     else:

/usr/lib/python2.3/site-packages/numarray/generic.py in _take(self,
indices, **keywds)
    795     def _take(self, indices, **keywds):
    796         indices = list(indices)
--> 797         impliedShape, N = _takeShape(self, indices)
    798         result = self._clone(shape=impliedShape)
    799         indices = self._fix_pt_indices(indices)

/usr/lib/python2.3/site-packages/numarray/generic.py in
_takeShape(scattered, indexArrays)
    133         nShape = scattered._shape[-leftOver:]
    134         if scattered.isbyteswapped() or not
scattered.iscontiguous():
--> 135             raise ValueError("Invalid destination array:
partial indices require contiguous non-byteswapped destination")
    136     else:
    137         N = scattered._itemsize

ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

On Thu, 2004-04-15 at 13:06, Faheem Mitha wrote:
> On Thu, 15 Apr 2004 05:13:29 +0000 (UTC), Faheem Mitha
> <fa...@em...> wrote:
> 
> > Hi,
> >
> > I'm trying to extract subarrays of a character array using take(), but
> > getting mysterious errors, for example:
> >
> > ValueError: Invalid destination array: partial indices require
> > contiguous non-byteswapped destination
> >
> > I am thinking that perhaps this operation is not supported for
> > character arrays, and I should convert my character array to a numeric
> > array instead (the strings are all of length 1).

This is a bug.  I logged it on source forge.

> 
> I read this again and realised this does not give any specific
> information, and is not very coherent. Got to avoid those late night
> postings. So here are the details. If I set axis = 0 there is no
> problem. I also don't have this problem with numerical arrays.
> 
> I am sure there is a simple reason for this, and hopefully a simple
> workaround.

Try this for a workaround:

s.swapaxes(0,1)
s = s.copy()
r = numarray.take(s, (1,2))
r.swapaxes(0,1)

Regards,
Todd

-- 
Todd Miller <jm...@st...>

Re: [Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004, Todd Miller wrote:

> Try this for a workaround:
>
> s.swapaxes(0,1)
> s = s.copy()
> r = numarray.take(s, (1,2))
> r.swapaxes(0,1)

I was trying

In [50]: ts = numarray.transpose(s)

In [51]: numarray.take(ts,(1,2),axis=0)
...
ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

In [52]: tscopy = numarray.transpose(s).copy()

In [53]: numarray.take(tscopy,(1,2),axis=0)
Out[53]:
CharArray([['b', 'e', 'h'],
           ['c', 'f', 'i']])

Is this the same or a related bug? Thanks for the reply and info.

                                                        Faheem.

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

On Thu, 2004-04-15 at 16:17, Faheem Mitha wrote:
> On Thu, 15 Apr 2004, Todd Miller wrote:
> 
> > Try this for a workaround:
> >
> > s.swapaxes(0,1)
> > s = s.copy()
> > r = numarray.take(s, (1,2))
> > r.swapaxes(0,1)
> 
> I was trying
> 
> In [50]: ts = numarray.transpose(s)
> 
> In [51]: numarray.take(ts,(1,2),axis=0)
> ...
> ValueError: Invalid destination array: partial indices require
> contiguous non-byteswapped destination
> 
> In [52]: tscopy = numarray.transpose(s).copy()
> 
> In [53]: numarray.take(tscopy,(1,2),axis=0)
> Out[53]:
> CharArray([['b', 'e', 'h'],
>            ['c', 'f', 'i']])
> 
> Is this the same or a related bug? 

Same bug.  Making a copy of the transposed array ensures that it (the
copy) is contiguous.

Regards,
Todd

-- 
Todd Miller <jm...@st...>

Re: [Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004, Todd Miller wrote:

> On Thu, 2004-04-15 at 16:17, Faheem Mitha wrote:
> > On Thu, 15 Apr 2004, Todd Miller wrote:
> >
> > > Try this for a workaround:
> > >
> > > s.swapaxes(0,1)
> > > s = s.copy()
> > > r = numarray.take(s, (1,2))
> > > r.swapaxes(0,1)
> >
> > I was trying
> >
> > In [50]: ts = numarray.transpose(s)
> >
> > In [51]: numarray.take(ts,(1,2),axis=0)
> > ...
> > ValueError: Invalid destination array: partial indices require
> > contiguous non-byteswapped destination
> >
> > In [52]: tscopy = numarray.transpose(s).copy()
> >
> > In [53]: numarray.take(tscopy,(1,2),axis=0)
> > Out[53]:
> > CharArray([['b', 'e', 'h'],
> >            ['c', 'f', 'i']])
> >
> > Is this the same or a related bug?
>
> Same bug.  Making a copy of the transposed array ensures that it (the
> copy) is contiguous.

I am wondering if this bug affects boost-python as well. I got the same
behaviour when using boost.python's version of take.

Can you point to where in the source code the bug is? Thanks.

                                                           Faheem.

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

On Thu, 2004-04-15 at 16:40, Faheem Mitha wrote:
> On Thu, 15 Apr 2004, Todd Miller wrote:
> 
> > On Thu, 2004-04-15 at 16:17, Faheem Mitha wrote:
> > > On Thu, 15 Apr 2004, Todd Miller wrote:
> > >
> > > > Try this for a workaround:
> > > >
> > > > s.swapaxes(0,1)
> > > > s = s.copy()
> > > > r = numarray.take(s, (1,2))
> > > > r.swapaxes(0,1)
> > >
> > > I was trying
> > >
> > > In [50]: ts = numarray.transpose(s)
> > >
> > > In [51]: numarray.take(ts,(1,2),axis=0)
> > > ...
> > > ValueError: Invalid destination array: partial indices require
> > > contiguous non-byteswapped destination
> > >
> > > In [52]: tscopy = numarray.transpose(s).copy()
> > >
> > > In [53]: numarray.take(tscopy,(1,2),axis=0)
> > > Out[53]:
> > > CharArray([['b', 'e', 'h'],
> > >            ['c', 'f', 'i']])
> > >
> > > Is this the same or a related bug?
> >
> > Same bug.  Making a copy of the transposed array ensures that it (the
> > copy) is contiguous.
> 
> I am wondering if this bug affects boost-python as well. I got the same
> behaviour when using boost.python's version of take.

The bug shouldn't affect numerical arrays because there are different
underlying implementations of take.

> Can you point to where in the source code the bug is? Thanks.

I think the relevant modules are numarray.generic and numarray.ufunc. 
There's a support function which computes the shape of the take() result
in generic.py, _takeShape().  It's testing for a property,
contiguousness, which is not required by the generic take code but is
required by the numerical code.  

The fix may be to factor the exception out of _takeShape and put it into
the (numerical) ufunc.put() setup code somewhere. 

Regards,
Todd

-- 
Todd Miller <jm...@st...>

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

This problem is now fixed in CVS.

Regards,
Todd
-- 
Todd Miller <jm...@st...>