Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>I think this should be not too difficult to achieve and I'll try to explain
>why.
>
>When going from numarray to Numeric, numarray already have NA_InputArray
>C-API function that returns a well-behaved array. But strictly speaking, we
>don't even need a well-behaved array (this is a too restrictive condition)
>as both Numeric and numarray support discontiguous data. Even the byteorder
>should be not a problem, because, as Numeric itself has no such a property,
>we can create a Numeric array that is in native order as the result and
>byteswap the numarray object (if needed) before doing the conversion.
>
In-place byteswapping sounds like a bad idea to me.  What if the array 
is based upon a readonly buffer?  We've just started using these at 
STSCI because a readonly memory map imposes no load on the system swap 
file.  With a read only mapping,  the buffer itself has readonly pages; 
 these cannot be swapped in-place.

>So, non-alignment remains as the only issue that may cause a buffer copy
>during numarray ==> Numeric conversion. Is that correct?. 
>
I don't think so.

>If yes, it is
>possible to do a workaround about that, i.e. we can still get a Numeric from
>a numarray without copying the data in case of numarray misaligned objects?.
>  
>
I don't see how.  The primary source of misaligned arrays is numerical 
columns in recarrays.  It seems to me that if the data is misaligned, 
 you either have to copy it to someplace else which is aligned,  or 
teach the function which is going to process it how to access it 
byte-wise.  Only the former sounds feasible to me.

>Regarding to going in the other sense (ie. Numeric ==> numarray), as
>numarray supports discontiguity, misalignment and byteswapped data, this
>conversion should not imply a data buffer copy at all. 
>  
>
This sounds correct.  

>Once we have a pointer to the data buffer, it is only a matter of
>wrapping a Numeric or numarray object around it getting this info from the
>original object, and returning the new object as a result.
>
>All in all, this conversion *seems* to be not a too difficult task.
>  
>
It seems straightforward in principle,  but the memory management issues 
seem a little tricky to me.   It's easy to get buffers from numarrays, 
and create numarrays from buffers.  I guess we need a module which does 
the same for Numeric.  

There are two easy ways to "get a buffer" from a Numeric array:

1.  Wrap the Numeric data in a buffer object.
2.  Add support for the buffer API to the Numeric object.

Off hand,  I'm not sure which is better,  although (1) is less intrusive 
to Numeric and I suppose is the place to start.  This should be easy.

But,  I'm not sure how to create a Numeric array from a buffer.  It's 
easy to get the data pointer from a buffer, and to construct a Numeric 
array from a data pointer,   but we also need a way to stash the pointer 
to the buffer object.    I don't like the idea of modifying Numeric's 
PyArrayObject.  

>Making such a conversion functions (in C, but also having Python
>counterparts) available might represent to open the door to a co-existence
>of Numeric and numarray objects in the same program, and that would easy the
>numarray deployment in existing Numeric software.
>
>Comments?
>  
>
All in all,  I think this is a great idea which would really boost 
interoperability.  I wish there was a simpler approach which required no 
modifications to Numeric.

Todd