numpy-discussion

[Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

Hi,

Anybody is aware of any function (either in C or Python or a mixture of
both) to easily convert Numerical Python arrays from/to numarray arrays?

I mean, I would like to use such a funtion that, without having to copy
element by element all the data, be able to copy the data buffer (or even
use the same if possible at all) from one object to the other.

Thanks,

--=20
Francesc Alted

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Sebastian H. <ha...@ms...>]

Hi,
I think this is actually quite related to my post from Friday:
[Numpy-discussion] make C array accessible to python without copy

-> So, to reformulate: Who hold actually the array data in memory? Or: where
gets the memory allocated and where/how many pointers to that exist?    I
understood the answer that Todd Miller gave, that there is such a thing as a
"buffer object" that does all the work, so then: one would just have to take
that and build a "new" numarray or Numeric structure around it  (referring
to the Subject of this email)   or  (in the case of my Friday-email)  just
have that "buffer object" point to a different memory space (that got
already allocated by the C-program) .

Agree ? (Did I get it right?)

Sebastian Haase

----- Original Message -----

From: "Francesc Alted" <fa...@op...>
To: <num...@li...>
Sent: Tuesday, January 21, 2003 8:44 AM
Subject: [Numpy-discussion] Conversion functions between Numeric and
numarray arrays?


Hi,

Anybody is aware of any function (either in C or Python or a mixture of
both) to easily convert Numerical Python arrays from/to numarray arrays?

I mean, I would like to use such a funtion that, without having to copy
element by element all the data, be able to copy the data buffer (or even
use the same if possible at all) from one object to the other.

Thanks,

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

A Dimarts 21 Gener 2003 19:41, Sebastian Haase va escriure:
> Hi,
> I think this is actually quite related to my post from Friday:
> [Numpy-discussion] make C array accessible to python without copy
>
> -> So, to reformulate: Who hold actually the array data in memory? Or:
> where gets the memory allocated and where/how many pointers to that exi=
st?=20
>   I understood the answer that Todd Miller gave, that there is such a t=
hing
> as a "buffer object" that does all the work, so then: one would just ha=
ve
> to take that and build a "new" numarray or Numeric structure around it=20
> (referring to the Subject of this email)   or  (in the case of my
> Friday-email)  just have that "buffer object" point to a different memo=
ry
> space (that got already allocated by the C-program) .
>
> Agree ? (Did I get it right?)

Well, so so. I think the buffer object is a property of numarray objects,
not Numeric objects. So, in the numarray =3D=3D> Numeric conversion proce=
ss you
may need to access the internals of the buffer (for example by using the
high level numarray C-API) and manage to obtain a data buffer (in the C
sense, not an object) that can be used to build the Numeric object (with =
the
help of the numarray object metadata). The opposite way needs something
similar but with inverted roles. See my previous message for a more in-de=
pth
explanation.

I think the conversion (without copying) is not a difficult process, but =
no
so-easy like that.

Well, I'm just a newcomer to numarray and my opinions about that may
perfectly be completely wrong, of course. Take them with caution!.

--=20
Francesc Alted

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

A Dimarts 21 Gener 2003 17:26, v=E0reu escriure:
> Francesc Alted wrote:
> >Anybody is aware of any function (either in C or Python or a mixture o=
f
> >both) to easily convert Numerical Python arrays from/to numarray array=
s?
>
> I think you should look at numarray.fromlist() and NumArray.tolist().  =
I
> think fromlist() will work on a nested sequence object,  and hence a
> Numeric array.

Yeah, I knew that, but I was looking for something more optimal.

>
> >I mean, I would like to use such a funtion that, without having to cop=
y
> >element by element all the data, be able to copy the data buffer (or e=
ven
> >use the same if possible at all) from one object to the other.
>
> I have not looked at this yet;   it's a very good question.  Note that
> going from numarray to Numeric there are issues with making the buffer
> well-behaved.

I think this should be not too difficult to achieve and I'll try to expla=
in
why.

When going from numarray to Numeric, numarray already have NA_InputArray
C-API function that returns a well-behaved array. But strictly speaking, =
we
don't even need a well-behaved array (this is a too restrictive condition=
)
as both Numeric and numarray support discontiguous data. Even the byteord=
er
should be not a problem, because, as Numeric itself has no such a propert=
y,
we can create a Numeric array that is in native order as the result and
byteswap the numarray object (if needed) before doing the conversion.

So, non-alignment remains as the only issue that may cause a buffer copy
during numarray =3D=3D> Numeric conversion. Is that correct?. If yes, it =
is
possible to do a workaround about that, i.e. we can still get a Numeric f=
rom
a numarray without copying the data in case of numarray misaligned object=
s?.

Regarding to going in the other sense (ie. Numeric =3D=3D> numarray), as
numarray supports discontiguity, misalignment and byteswapped data, this
conversion should not imply a data buffer copy at all.=20

Once we have a pointer to the data buffer, it is only a matter of
wrapping a Numeric or numarray object around it getting this info from th=
e
original object, and returning the new object as a result.

All in all, this conversion *seems* to be not a too difficult task.

Making such a conversion functions (in C, but also having Python
counterparts) available might represent to open the door to a co-existenc=
e
of Numeric and numarray objects in the same program, and that would easy =
the
numarray deployment in existing Numeric software.

Comments?

--=20
Francesc Alted

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>I think this should be not too difficult to achieve and I'll try to explain
>why.
>
>When going from numarray to Numeric, numarray already have NA_InputArray
>C-API function that returns a well-behaved array. But strictly speaking, we
>don't even need a well-behaved array (this is a too restrictive condition)
>as both Numeric and numarray support discontiguous data. Even the byteorder
>should be not a problem, because, as Numeric itself has no such a property,
>we can create a Numeric array that is in native order as the result and
>byteswap the numarray object (if needed) before doing the conversion.
>
In-place byteswapping sounds like a bad idea to me.  What if the array 
is based upon a readonly buffer?  We've just started using these at 
STSCI because a readonly memory map imposes no load on the system swap 
file.  With a read only mapping,  the buffer itself has readonly pages; 
 these cannot be swapped in-place.

>So, non-alignment remains as the only issue that may cause a buffer copy
>during numarray ==> Numeric conversion. Is that correct?. 
>
I don't think so.

>If yes, it is
>possible to do a workaround about that, i.e. we can still get a Numeric from
>a numarray without copying the data in case of numarray misaligned objects?.
>  
>
I don't see how.  The primary source of misaligned arrays is numerical 
columns in recarrays.  It seems to me that if the data is misaligned, 
 you either have to copy it to someplace else which is aligned,  or 
teach the function which is going to process it how to access it 
byte-wise.  Only the former sounds feasible to me.

>Regarding to going in the other sense (ie. Numeric ==> numarray), as
>numarray supports discontiguity, misalignment and byteswapped data, this
>conversion should not imply a data buffer copy at all. 
>  
>
This sounds correct.  

>Once we have a pointer to the data buffer, it is only a matter of
>wrapping a Numeric or numarray object around it getting this info from the
>original object, and returning the new object as a result.
>
>All in all, this conversion *seems* to be not a too difficult task.
>  
>
It seems straightforward in principle,  but the memory management issues 
seem a little tricky to me.   It's easy to get buffers from numarrays, 
and create numarrays from buffers.  I guess we need a module which does 
the same for Numeric.  

There are two easy ways to "get a buffer" from a Numeric array:

1.  Wrap the Numeric data in a buffer object.
2.  Add support for the buffer API to the Numeric object.

Off hand,  I'm not sure which is better,  although (1) is less intrusive 
to Numeric and I suppose is the place to start.  This should be easy.

But,  I'm not sure how to create a Numeric array from a buffer.  It's 
easy to get the data pointer from a buffer, and to construct a Numeric 
array from a data pointer,   but we also need a way to stash the pointer 
to the buffer object.    I don't like the idea of modifying Numeric's 
PyArrayObject.  

>Making such a conversion functions (in C, but also having Python
>counterparts) available might represent to open the door to a co-existence
>of Numeric and numarray objects in the same program, and that would easy the
>numarray deployment in existing Numeric software.
>
>Comments?
>  
>
All in all,  I think this is a great idea which would really boost 
interoperability.  I wish there was a simpler approach which required no 
modifications to Numeric.

Todd