numpy-discussion

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>Hi,
>
>I'm trying to make a C array from a Numeric "c" (Character) typecode array
>using the high level call:
>
>NA_InputArray(PyObject *numarray, NumarrayType t, int requires)
>
Unified handling of character arrays and numeric arrays doesn't exist 
yet in numarray.  There is no C-API for the chararray module because we 
haven't needed one.  But CharArrays are NDArrays and have attributes 
stored in PyArrayObjects just like numarrays.

>with no success.
>
>As I have been able to access all the other types (i.e.
>'1','b','s','i','l','f','d') successfully, perhaps character type is not
>supported?
>
>In the NumarrayType enum, there is no tChar, but I've tried tUInt8 and tAny
>as the value for NumarrayType parameter, but both choices issues the same
>error:
>
>Traceback (most recent call last):
>  File "table-tree2.py", line 77, in ?
>    h5file.createArray('/columns', 'name', array(names), "Name column")
>  File "/home/falted/PyTables/pytables-0.3/tables/File.py", line 400, in 
>createArray
>    setattr(group, name, object)
>  File "/home/falted/PyTables/pytables-0.3/tables/Group.py", line 355, in 
>__setattr__
>    value._f_putObjectInTree(name, self)
>  File "/home/falted/PyTables/pytables-0.3/tables/Leaf.py", line 71, in 
>_f_putObjectInTree
>    self.create()
>  File "/home/falted/PyTables/pytables-0.3/tables/Array.py", line 83, in 
>create
>    self.createArray(self.object, self.title)
>  File "/home/falted/PyTables/pytables-0.3/src/hdf5Extension.pyx", line 913, 
>in createArray
>    array = NA_InputArray(arr, numfmt2[arr.typecode()], C_ARRAY)
>libnumarray.error: getShape: sequence object nested more than MAXDIM deep.
>
NA_InputArray was intended to accept non-numeric sequences.  It could 
report this better...

>although I was passing only a Numeric 'c' with a rather small shape (10,16).
>
>I just want to access the buffer data, and the shape of this object from C
>(well, I'm actually using Pyrex, but I think this is not important). Is that
>possible by only using numarray C calls?
>
Look at Lib/chararray.py and Src/_chararraymodule.c.

If you can handle using a CharArray or RawCharArray, try:

1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in 
the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.

2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.

3. shape, strides, and itemsize should be directly accessible from the 
PyArrayObject.

CharArray has some extra stripping and padding semantics; these are lazy 
and hence absent without extra care in C.  RawCharArray has none.

CharArrays are really arrays of fixed length strings of bytes.  The 
string length is defined by the array itemsize.

>Thanks,
>
>  
>
Todd

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Francesc A. <fa...@op...>]

A Dissabte 18 Gener 2003 17:41, Todd Miller va escriure:
> >I just want to access the buffer data, and the shape of this object fr=
om C
> >(well, I'm actually using Pyrex, but I think this is not important). I=
s
> > that possible by only using numarray C calls?
>
> Look at Lib/chararray.py and Src/_chararraymodule.c.
>
> If you can handle using a CharArray or RawCharArray, try:
>
> 1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in
> the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.
>
> 2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.
>
> 3. shape, strides, and itemsize should be directly accessible from the
> PyArrayObject.

Ok. I'll try to do that.

>
> CharArray has some extra stripping and padding semantics; these are laz=
y
> and hence absent without extra care in C.  RawCharArray has none.
>

By the way, is it safe to assume that CharArray objects are contiguous? o=
r
RawCharArray?. The same question goes for RecArray objects. Or it is alwa=
ys
convenient to check with iscontiguous() method if they are or not?. In ca=
se
these objects can be non-contiguous, I guess there's still not a function
like NA_InputArray that works with CharArray or RecArray objects in order=
 to
obtain well-behaved objects. Is that true?

I think it would be possible to me to include support for numarray object=
s
in next release of PyTables. Thanks!,

--=20
Francesc Alted

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dissabte 18 Gener 2003 17:41, Todd Miller va escriure:
>  
>
>>>I just want to access the buffer data, and the shape of this object from C
>>>(well, I'm actually using Pyrex, but I think this is not important). Is
>>>that possible by only using numarray C calls?
>>>      
>>>
>>Look at Lib/chararray.py and Src/_chararraymodule.c.
>>
>>If you can handle using a CharArray or RawCharArray, try:
>>
>>1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in
>>the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.
>>
>>2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.
>>
>>3. shape, strides, and itemsize should be directly accessible from the
>>PyArrayObject.
>>    
>>
>
>Ok. I'll try to do that.
>
>  
>
>>CharArray has some extra stripping and padding semantics; these are lazy
>>and hence absent without extra care in C.  RawCharArray has none.
>>
>>    
>>
>
>By the way, is it safe to assume that CharArray objects are contiguous? or
>RawCharArray?.
>
Mostly no.   Each fixed length element is stored as a contiguous 
sequence of bytes.  Anything goes for the rest,  so you need to look at 
the strides arrays and byteoffset.

>The same question goes for RecArray objects. 
>
No.  It's possible to select every 10th record, for instance, in a 
slice.  I believe the resulting decimated array would be a discontiguous 
view of the original.  

>Or it is always
>convenient to check with iscontiguous() method if they are or not?.
>
I'm not even certain the method works correctly for chararray and 
recarray.  

I think the portion of chararray that has been written in C considers 
array strides.
recarray is pure python.  In both cases,  I think I'd just forget about 
contiguity and use the strides arrays.

> In case
>these objects can be non-contiguous, I guess there's still not a function
>like NA_InputArray that works with CharArray or RecArray objects in order to
>obtain well-behaved objects. Is that true?
>
True.  But neither recarray nor chararray really has behavedness 
problems like misalignment,
byteswapping, or type conversion.  I think contiguity is the only issue, 
and that is solved
just by calling .copy().  You might argue that  records contain 
byteswapped and misaligned fields.   I don't have an immediate answer to 
that.

My preference is to use strides and forget about contiguity,  but you 
could also make contiguous copies simply.  Noone I'm aware of has yet 
tried access to misbehaved records in C.

>
>I think it would be possible to me to include support for numarray objects
>in next release of PyTables. 
>
Great!

>Thanks!,
>  
>

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Francesc A. <fa...@op...>]

A Dissabte 18 Gener 2003 21:12, Todd Miller va escriure:
> >By the way, is it safe to assume that CharArray objects are contiguous=
? or
> >RawCharArray?.
>
> Mostly no.   Each fixed length element is stored as a contiguous
> sequence of bytes.  Anything goes for the rest,  so you need to look at
> the strides arrays and byteoffset.
>
> >The same question goes for RecArray objects.
>
> No.  It's possible to select every 10th record, for instance, in a
> slice.  I believe the resulting decimated array would be a discontiguou=
s
> view of the original.
>
> >Or it is always
> >convenient to check with iscontiguous() method if they are or not?.
>
> I'm not even certain the method works correctly for chararray and
> recarray.

Well, during my tests with numarray 0.4, iscontiguous() seems to work wel=
l,
both for chararrays and recarrays.

> In both cases,  I think I'd just forget about
> contiguity and use the strides arrays.

Yeah, but I still want to use iscontiguous() method just to speed-up a bi=
t
the code.

> You might argue that  records contain
> byteswapped and misaligned fields.   I don't have an immediate answer t=
o
> that.

Exactly, I am pondering how to deal with HDF5 objects coming from machine=
s
with a different endianess (misalignment is not a problem in my case) tha=
n
the local machine. But I think I can manage that by creating recarrays
buffers with the byteorder parameter set appropriately during the HDF5 ta=
ble
reads. Then, all the data can be read correctly because numarray will
byteswap the data whenever this recarray will be accessed.

Moreover, if this object is to be used frequently, I can speed-up the acc=
ess
to this recarray by byteswapping the columns (as arrays) using their
byteswap() method. In the future it would be nice to provide a generica
byteswap method for recarrays.

Thanks,

--=20
Francesc Alted