Menu
▾
▴
numpy-discussion
|
From: Ville H. <ville@e.math.helsinki.fi> - 2004-04-22 12:51:46
|
Dear all, I would like to turn a large C array allocated by a library into a Numarray object without copying the contents. In other words I have a given array of double precision floats and I want to get a Numarray object, where the data pointer points to this array and no additional memory is allocated. Naturally when the reference count of the Numarray object gets back to zero, the object is freed, but the array itself is left as a nuisance for the C code. Is there a standard way of doing this? Thanks in advance, -- Ville |
|
From: Todd M. <jm...@st...> - 2004-04-22 13:10:17
|
On Thu, 2004-04-22 at 08:51, Ville Hakulinen wrote: > Dear all, > > I would like to turn a large C array allocated by a library into a > Numarray object without copying the contents. In other words I have > a given array of double precision floats and I want to get a Numarray > object, where the data pointer points to this array and no additional > memory is allocated. Naturally when the reference count of the Numarray > object gets back to zero, the object is freed, but the array itself > is left as a nuisance for the C code. > > Is there a standard way of doing this? No, not yet. You're not the first person to ask for this but I'd appreciate it if you'd explain why you need it. So far, not having that feature is a conscious omission to keep complexity down. Regards, Todd -- Todd Miller <jm...@st...> |
|
From: Craig R. <cra...@la...> - 2004-04-22 14:28:29
|
On Apr 22, 2004, at 7:10 AM, Todd Miller wrote: >> I would like to turn a large C array allocated by a library into a >> Numarray object without copying the contents. In other words I have >> a given array of double precision floats and I want to get a Numarray >> object, where the data pointer points to this array and no additional >> memory is allocated. Naturally when the reference count of the >> Numarray >> object gets back to zero, the object is freed, but the array itself >> is left as a nuisance for the C code. >> >> Is there a standard way of doing this? > > No, not yet. You're not the first person to ask for this but I'd > appreciate it if you'd explain why you need it. So far, not having > that feature is a conscious omission to keep complexity down. The reason I need this is two fold: 1. I want to export to Python existing Fortran arrays. If you want Fortran and Python to share array memory, you can't always expect Fortran programmers to use memory allocated in Python. In fact, since there is no standards conforming way to associate a Fortran array pointer with Python memory, you have to do the opposite, associate a Python array with Fortran memory. 2. In the CoArray Python module, we MUST use special memory allocators that are parallel aware. Because of these requirements, our only choice is to use Numeric. Regards, Craig |
|
From: Sebastian H. <ha...@ms...> - 2004-04-22 16:46:03
|
Hi Craig,
I am sharing arrays between C and Python numarray (w/o any copying) for more
than a year by now - please check my postings and (mostly) Todd's replys on
this list. The code looks mostly like this:
<code sniplet>
int shape[4]= { n,nCams,pic_ny,pic_nx };
PyObject *obj, *obj2;
obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(),
theRing.bufferSize());
obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0,
NA_ByteOrder(), 1, 1/*writable*/) ;
wxASSERT(obj != NULL);
wxASSERT(obj2 != NULL);
PyDict_SetItemString(globals, "ring", obj2);
Py_XDECREF(obj);
Py_XDECREF(obj2);
</code sniplet>
It even works with records arrays!
Luckily my arrays stay (mostly) around until the program quits.
So I also don't have a good answer for 'who should free the memory?'
As far as numarray/Python concerns - once it's reference counter goes to zero
- a 'delete[]' would be needed if the array was allocated with a C++ 'new'
but a 'free()' is required if the memory got "malloc()'ed" in C.
Our internal discussion here found that the only real solution to this might
be to extend numarray's object structure by a functions pointer
('callToKill') and also an argument pointer (the array should be delete'd or
free'd).
But then we felt this might be to much to ask (to much of a change / overhead)
that this proposal would not get accepted - so we choose to not bring it up.
Please comment,
Regards
Sebastian Haase
On Thursday 22 April 2004 07:28 am, Craig Rasmussen wrote:
> On Apr 22, 2004, at 7:10 AM, Todd Miller wrote:
> >> I would like to turn a large C array allocated by a library into a
> >> Numarray object without copying the contents. In other words I have
> >> a given array of double precision floats and I want to get a Numarray
> >> object, where the data pointer points to this array and no additional
> >> memory is allocated. Naturally when the reference count of the
> >> Numarray
> >> object gets back to zero, the object is freed, but the array itself
> >> is left as a nuisance for the C code.
> >>
> >> Is there a standard way of doing this?
> >
> > No, not yet. You're not the first person to ask for this but I'd
> > appreciate it if you'd explain why you need it. So far, not having
> > that feature is a conscious omission to keep complexity down.
>
> The reason I need this is two fold:
>
> 1. I want to export to Python existing Fortran arrays. If you want
> Fortran and Python to share array memory, you can't always expect
> Fortran programmers to use memory allocated in Python. In fact,
> since there is no standards conforming way to associate a Fortran
> array pointer with Python memory, you have to do the opposite,
> associate a Python array with Fortran memory.
>
> 2. In the CoArray Python module, we MUST use special memory
> allocators that are parallel aware.
>
> Because of these requirements, our only choice is to use Numeric.
>
> Regards,
> Craig
|
|
From: Todd M. <jm...@st...> - 2004-04-22 16:57:31
|
On Thu, 2004-04-22 at 11:47, Paul F. Dubois wrote: > This is what PyArray_FromDimsAndData was for, and I believe it is fairly > heavily used. That's good to know. I was just discussing with Perry how to expand the C-API to support this stuff. Perhaps we don't have to. > Todd Miller wrote: > > On Thu, 2004-04-22 at 08:51, Ville Hakulinen wrote: > > > >>Dear all, > >> > >>I would like to turn a large C array allocated by a library into a > >>Numarray object without copying the contents. In other words I have > >>a given array of double precision floats and I want to get a Numarray > >>object, where the data pointer points to this array and no additional > >>memory is allocated. Naturally when the reference count of the Numarray > >>object gets back to zero, the object is freed, but the array itself > >>is left as a nuisance for the C code. > >> > >>Is there a standard way of doing this? > > > > > > No, not yet. You're not the first person to ask for this but I'd > > appreciate it if you'd explain why you need it. So far, not having > > that feature is a conscious omission to keep complexity down. > > > > Regards, > > Todd > > -- Todd Miller <jm...@st...> |
|
From: Ville H. <ville@e.math.helsinki.fi> - 2004-04-22 18:23:44
|
On Thu, Apr 22, 2004 at 09:10:07AM -0400, Todd Miller wrote: > No, not yet. You're not the first person to ask for this but I'd > appreciate it if you'd explain why you need it. So far, not having > that feature is a conscious omission to keep complexity down. First I managed not to reply to numpy-discussion, so here we go again: We are building a Python interface to PETSc 2.2 (http://www-unix.mcs.anl.gov/petsc/petsc-2/) using SWIG. It would be very convenient to access the part of a PETSc vector that is local to a process(or) using Numarray. Other options would be to just write functions to access the part of a vector independently of Numarray or to figure out whether one could tell PETSc to use the memory area allocated by Numarray. -- Ville |
|
From: Craig R. <cra...@la...> - 2004-04-22 18:32:27
|
On Apr 22, 2004, at 10:46 AM, Sebastian Haase wrote: > obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), > theRing.bufferSize()); > obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, > NA_ByteOrder(), 1, 1/*writable*/) ; It looks like this will work. I thought Todd had been saying that this wasn't possible. Perhaps I did not phrase my question very well when I asked earlier. Thanks, Craig |
|
From: Todd M. <jm...@st...> - 2004-04-22 19:08:54
|
On Thu, 2004-04-22 at 14:32, Craig Rasmussen wrote: > On Apr 22, 2004, at 10:46 AM, Sebastian Haase wrote: > > > obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), > > theRing.bufferSize()); > > obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, > > NA_ByteOrder(), 1, 1/*writable*/) ; > > It looks like this will work. > It'll definitely work. I'd forgotten about it. > I thought Todd had been saying that this wasn't possible. Perhaps I > did not phrase my question very well when I asked earlier. Nope, your question was fine. I'm hopeful that PyArray_FromDimsAndData can be improved so that this becomes easier for numarray-1.0. Regards, Todd -- Todd Miller <jm...@st...> |
|
From: Craig R. <cra...@la...> - 2004-04-22 18:34:35
|
On Apr 22, 2004, at 10:57 AM, Todd Miller wrote: >> This is what PyArray_FromDimsAndData was for, and I believe it is >> fairly >> heavily used. > > That's good to know. I was just discussing with Perry how to expand > the > C-API to support this stuff. Perhaps we don't have to. So is there an equivalent NA_FromDimsAndData? I have been using PyArray_FromDimsAnd Data. Craig |
|
From: Todd M. <jm...@st...> - 2004-04-22 20:06:06
|
On Thu, 2004-04-22 at 14:34, Craig Rasmussen wrote: > On Apr 22, 2004, at 10:57 AM, Todd Miller wrote: > > >> This is what PyArray_FromDimsAndData was for, and I believe it is > >> fairly > >> heavily used. > > > > That's good to know. I was just discussing with Perry how to expand > > the > > C-API to support this stuff. Perhaps we don't have to. > > So is there an equivalent NA_FromDimsAndData? No. I was making the mistake of thinking there needed to be. > I have been using > PyArray_FromDimsAndData. That's fine and should basically work. Between numarray and Numeric there is currently the not-so-subtle difference that numarray copies the external data buffer and Numeric just refers to it. This is a documented difference, but my vote is for changing in favor of better functionality and Numeric compatibility. I'll probably just quietly change it unless there is a deluge of dissenters... -- Todd Miller <jm...@st...> |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X