Menu
▾
▴
numpy-discussion
|
From: Todd M. <jm...@st...> - 2003-07-15 19:31:43
|
I am adding arrays of Python objects to numarray and so I am curious about the uses people have found for Numeric's object arrays. If you have found Numeric's object arrays useful, please tell us about what you used them for so that we can make certain that numarray can satisfy the same need. Thanks, Todd -- Todd Miller jm...@st... STSCI / ESS / SSB |
|
From: Konrad H. <hi...@cn...> - 2003-07-16 09:26:55
|
On Tuesday 15 July 2003 21:34, Todd Miller wrote: > I am adding arrays of Python objects to numarray and so I am curious > about the uses people have found for Numeric's object arrays. If you > have found Numeric's object arrays useful, please tell us about what > you used them for so that we can make certain that numarray can satisfy > the same need. I have used them several times in order to be able to use array operation= s on=20 number-like objects, e.g. multiple precision numbers, quaternions, automa= tic=20 derivatives, polynomials, etc. Konrad. --=20 -------------------------------------------------------------------------= ------ Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------= ------ |
|
From: Todd M. <jm...@st...> - 2003-07-16 16:29:31
|
On Wed, 2003-07-16 at 05:26, Konrad Hinsen wrote: > On Tuesday 15 July 2003 21:34, Todd Miller wrote: > > I am adding arrays of Python objects to numarray and so I am curious > > about the uses people have found for Numeric's object arrays. If you > > have found Numeric's object arrays useful, please tell us about what > > you used them for so that we can make certain that numarray can satisfy > > the same need. > > I have used them several times in order to be able to use array operations on > number-like objects, e.g. multiple precision numbers, quaternions, automatic > derivatives, polynomials, etc. OK. Right now the prototype supports array structural operations (indexing, slicing, array indexing, copies, views, transpose...) and element-wise operator application (so when you add two object arrays, the corresponding objects are added using the number protocol). Support for universal functions (calling numarray.add(a,b) rather than a+b) and the C-API for object arrays is still missing. Did these matter to your application of object arrays? -- Todd Miller <jm...@st...> |
|
From: Konrad H. <hi...@cn...> - 2003-07-16 17:35:18
|
On Wednesday 16 July 2003 18:29, you wrote: > Support for universal functions (calling numarray.add(a,b) rather than > a+b) and the C-API for object arrays is still missing. Did these matte= r > to your application of object arrays? Not add(a, b), but add.reduce(a) as well as things like log(a). Konrad. --=20 -------------------------------------------------------------------------= ------ Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------= ------ |
|
From: Todd M. <jm...@st...> - 2003-07-16 17:49:55
|
On Wed, 2003-07-16 at 13:34, Konrad Hinsen wrote: > On Wednesday 16 July 2003 18:29, you wrote: > > > Support for universal functions (calling numarray.add(a,b) rather than > > a+b) and the C-API for object arrays is still missing. Did these matter > > to your application of object arrays? > > Not add(a, b), but add.reduce(a) as well as things like log(a). OK. That makes sense. I have a fair amount of work left to do... > Konrad. -- Todd Miller <jm...@st...> |
|
From: Tim C. <tc...@op...> - 2003-07-16 21:45:28
|
On Wed, 2003-07-16 at 05:34, Todd Miller wrote: > I am adding arrays of Python objects to numarray and so I am curious > about the uses people have found for Numeric's object arrays. If you > have found Numeric's object arrays useful, please tell us about what > you used them for so that we can make certain that numarray can satisfy > the same need. We use NumPy to store vectors (rank-1 arrays) of numbers representing columns in a dataset. The NumPy arrays, which are large and numerous)=20 are memory-mapped (using an extension) to disc to conserve real memory. However, in some vectors (columns) we need to store variable-length, and in others, variable length sequences of integers or floats (and possibly even sets in the future). NumPy's object arrays are more memory-efficient that Python lists of lists or lists of strings from these purposes, and of course they support NumPy functions such as take(), which makes life simpler. But we haven't been able to memory-map these object arrays, which is a problem. Is there any prospect of numarray supporting memory-mapped arrays of sequences/strings? I know that is a big ask! We have an extension module which stores variable length blobs in a single memory-mapped file which might be useful - the code could be made available to the numarray project, I think. We also use MA extensively (because in the health care domain life is full of missing data) - I'll jot down some thoughts on how MA could be improved in the next few days. --=20 Tim C PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere or at http://members.optushome.com.au/tchur/pubkey.asc Key fingerprint =3D 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0 |
|
From: Todd M. <jm...@st...> - 2003-07-16 22:36:22
|
On Wed, 2003-07-16 at 17:43, Tim Churches wrote: > On Wed, 2003-07-16 at 05:34, Todd Miller wrote: > > I am adding arrays of Python objects to numarray and so I am curious > > about the uses people have found for Numeric's object arrays. If you > > have found Numeric's object arrays useful, please tell us about what > > you used them for so that we can make certain that numarray can satisfy > > the same need. > > We use NumPy to store vectors (rank-1 arrays) of numbers representing > columns in a dataset. The NumPy arrays, which are large and numerous) > are memory-mapped (using an extension) to disc to conserve real memory. > However, in some vectors (columns) we need to store variable-length, and > in others, variable length sequences of integers or floats (and possibly > even sets in the future). NumPy's object arrays are more > memory-efficient that Python lists of lists or lists of strings from Well, right now the prototype actually uses a single list internally as the object store; still, we might beat out lists of lists by a small margin. > these purposes, and of course they support NumPy functions such as > take(), which makes life simpler. The prototype currently uses common code for put/take on strings, object arrays, and soon record arrays. The common code is currently Python prototype. Numarray numeric arrays use specialized C-code for speed. > But we haven't been able to memory-map > these object arrays, which is a problem. Is there any prospect of > numarray supporting memory-mapped arrays of sequences/strings? numarray supports arrays of fixed length strings with its chararray module. The default chararray string stripping and padding functions blank fill unused space and give the appearance of variable length strings. The data buffers of all of numarray's classes which represent primitive data items (numbers, strings, records) can be memory mapped. I think however that memory mapping sequences or arbitrary Python objects isn't going to happen in numarray any time soon; it sounds too much like object persistence. > I know > that is a big ask! We have an extension module which stores variable > length blobs in a single memory-mapped file which might be useful - the > code could be made available to the numarray project, I think. I don't understand the difference between your module and Python's mmap. > > We also use MA extensively (because in the health care domain life is > full of missing data) - I'll jot down some thoughts on how MA could be > improved in the next few days. I'd be very interested in hearing your thoughts on improving MA. -- Todd Miller <jm...@st...> |
|
From: Dave C. <dj...@ob...> - 2003-07-17 01:13:22
|
>>>>> "Todd" == Todd Miller <jm...@st...> writes: Todd> I think however that memory mapping sequences or arbitrary Todd> Python objects isn't going to happen in numarray any time soon; Todd> it sounds too much like object persistence. It is used to minimize the cost of huge datasets. >> I know that is a big ask! We have an extension module which stores >> variable length blobs in a single memory-mapped file which might be >> useful - the code could be made available to the numarray project, >> I think. Todd> I don't understand the difference between your module and Todd> Python's mmap. The code could use probably the Python mmap module. It would be difficult because all access to the Python mmap object is via Python. A CObject exported API in the Python mmap module would be very useful. - Dave -- http://www.object-craft.com.au |
|
From: Todd M. <jm...@st...> - 2003-07-17 10:29:02
|
On Wed, 2003-07-16 at 21:13, Dave Cole wrote: > >>>>> "Todd" == Todd Miller <jm...@st...> writes: > > Todd> I think however that memory mapping sequences or arbitrary > Todd> Python objects isn't going to happen in numarray any time soon; > Todd> it sounds too much like object persistence. > > It is used to minimize the cost of huge datasets. That's what we use it for too... but only for primitive data types. > > >> I know that is a big ask! We have an extension module which stores > >> variable length blobs in a single memory-mapped file which might be > >> useful - the code could be made available to the numarray project, > >> I think. > > Todd> I don't understand the difference between your module and > Todd> Python's mmap. > > The code could use probably the Python mmap module. It would be > difficult because all access to the Python mmap object is via Python. > > A CObject exported API in the Python mmap module would be very useful. > What do you want to get via the CObject API? Since any mmap can be treated as a buffer, it's easy to get a pointer and size via the Python buffer API in C. Is there something else you need? > - Dave -- Todd Miller <jm...@st...> |
|
From: Dave C. <dj...@ob...> - 2003-07-17 11:25:44
|
>>>>> "Todd" == Todd Miller <jm...@st...> writes: Todd> I don't understand the difference between your module and Todd> Python's mmap. >> The code could use probably the Python mmap module. It would be >> difficult because all access to the Python mmap object is via >> Python. >> >> A CObject exported API in the Python mmap module would be very >> useful. >> Todd> What do you want to get via the CObject API? Since any mmap can Todd> be treated as a buffer, it's easy to get a pointer and size via Todd> the Python buffer API in C. Is there something else you need? Nope. I wonder why I didn't realise that... Too stupid I suppose. - Dave -- http://www.object-craft.com.au |
|
From: Todd M. <jm...@st...> - 2003-07-17 11:11:57
|
On Thu, 2003-07-17 at 06:44, Dave Cole wrote: > >>>>> "Todd" == Todd Miller <jm...@st...> writes: > > Todd> I don't understand the difference between your module and > Todd> Python's mmap. > > >> The code could use probably the Python mmap module. It would be > >> difficult because all access to the Python mmap object is via > >> Python. > >> > >> A CObject exported API in the Python mmap module would be very > >> useful. > >> > > Todd> What do you want to get via the CObject API? Since any mmap can > Todd> be treated as a buffer, it's easy to get a pointer and size via > Todd> the Python buffer API in C. Is there something else you need? > > Nope. I wonder why I didn't realise that... Too stupid I suppose. I doubt that or you wouldn't be using Python... The buffer API is just a little obscure. > > - Dave -- Todd Miller <jm...@st...> |
|
From: Francesc A. <fa...@op...> - 2003-07-24 07:29:03
|
Some time ago, talking about object arrays Todd Miller wrote: A Dijous 17 Juliol 2003 00:36, Todd Miller va escriure: > The prototype currently uses common code for put/take on strings, object > arrays, and soon record arrays. I am thinking on supporting multimensional arrays of record arrays soon in pytables. From your words, it seems that you are planning implementing this in the next release of numarray, isn't it? That would be nice. -- Francesc Alted |
|
From: Todd M. <jm...@st...> - 2003-07-24 12:08:14
|
On Thu, 2003-07-24 at 03:28, Francesc Alted wrote: > Some time ago, talking about object arrays Todd Miller wrote: > > A Dijous 17 Juliol 2003 00:36, Todd Miller va escriure: > > The prototype currently uses common code for put/take on strings, object > > arrays, and soon record arrays. > > I am thinking on supporting multimensional arrays of record arrays soon in > pytables. From your words, it seems that you are planning implementing this > in the next release of numarray, isn't it? That would be nice. Yes. The main feature of the next release will be support for arrays of Objects, similar to Numeric's typecode='O' arrays. > > -- > Francesc Alted > > -- Todd Miller jm...@st... STSCI / ESS / SSB |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X