numpy-discussion

[Numpy-discussion] Non-numerical info associated with sub-arrays

[Magnus L. H. <ma...@he...>]

I'm working on some two-dimensional tables of data, where some data
are numerical, while other aren't. I'd like to use numarray's
numerical capabilities with the numerical parts (columns) while
keeping the data in each row together. (I'm sure this generalizes to
more dimensions, and to sub-arrays in general, not just rows.)

It's not a hard problem, really, but the obvious solution--to keep
the other rows in separate arrays/lists and just juggle things
around--seems a bit clunky. I was just wondering if anyone had other
ideas (would it be practical to include all the data in a single array
somehow--I seem to recall that Numeric could have arbitrary object
arrays, but I'm not sure whether numarray supports this?) or perhaps
some hints on how to organize code around this? I wrote a small class
that wraps things up and works a bit lik R/S-plus's data frames; is
there some other more standard code out there for this sort of thing?
(It's a problem that crops up often in data processing of various
kinds...)

Thanks,

Magnus

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Tim C. <tc...@op...>]

On Fri, 2002-12-27 at 11:29, Magnus Lie Hetland wrote:
> I'm working on some two-dimensional tables of data, where some data
> are numerical, while other aren't. I'd like to use numarray's
> numerical capabilities with the numerical parts (columns) while
> keeping the data in each row together. (I'm sure this generalizes to
> more dimensions, and to sub-arrays in general, not just rows.)
> 
> It's not a hard problem, really, but the obvious solution--to keep
> the other rows in separate arrays/lists and just juggle things
> around--seems a bit clunky. I was just wondering if anyone had other
> ideas (would it be practical to include all the data in a single array
> somehow--I seem to recall that Numeric could have arbitrary object
> arrays, but I'm not sure whether numarray supports this?) or perhaps
> some hints on how to organize code around this? I wrote a small class
> that wraps things up and works a bit lik R/S-plus's data frames; is
> there some other more standard code out there for this sort of thing?
> (It's a problem that crops up often in data processing of various
> kinds...)

Have a look at the discussion on RecordArrays in this overview of
Numarray: http://stsdas.stsci.edu/numarray/DesignOverview.html

However, in the meantime, as you note, its not too hard to write a class
which emulates R/S-Plus data frames. Just store each column in its own
Numeric array of the appropriate type (which might be the PyObject
types, which can hold any Python object type), and have the wrapper
class implement __getitem__ etc to collect the relevant "rows" from each
column and return them as a complete row as a dict or a sequence. Not
that fast, but not slow either. You can implement a generator to allow
cursor-like traversal of the all the rows if you like. Happy to
collaborate on furthering this idea. By memory-mapping disc-based
versions of the  Numeric arrays, and using the BsdDb3 record number
database format for the string columns, you can even make a disc-based
"record array" which can be larger than available RAM+swap. I hope to
release code written under contract by Dave Cole (see
http://www.object-craft.com.au ) which illustrates this idea in the next
month or so (but I've been saying that to myself for a year or more...).

Tim C
 

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Magnus L. H. <ma...@he...>]

Tim Churches <tc...@op...>:
[snip]
> Have a look at the discussion on RecordArrays in this overview of
> Numarray: http://stsdas.stsci.edu/numarray/DesignOverview.html

Sounds interesting.

> However, in the meantime, as you note, its not too hard to write a class
> which emulates R/S-Plus data frames. Just store each column in its own
> Numeric array of the appropriate type

Yeah -- it's just that I'd like to keep a set of columns collected as
a two-dimensional array, to allow horizontal summing and the like.
(Not much more complicated, but an extra issue to address.)

> (which might be the PyObject
> types, which can hold any Python object type),

Hm. Yes. I can't seem to find these anymore. I seem to recall using
type='o' or something in Numeric, but I can't find the right type
objects now... (Guess I'm just reading the docs and dir(numeric)
poorly...) It would be nice if array(['foo']) just worked. Oh, well.

[snip]
> Happy to
> collaborate on furthering this idea.

That would be great (even though I don't really have any time to use
for this -- it's just a really tiny part of a small project I'm
working on :)

> By memory-mapping disc-based
> versions of the  Numeric arrays, and using the BsdDb3 record number
> database format for the string columns, you can even make a disc-based
> "record array" which can be larger than available RAM+swap.

Sounds quite useful, although quite similar to MetaKit. (I suppose I
could use some functions from numarray on columns in MetaKit... But
that might just be too weird -- and it would still just be a
collection of columns :])

[snip]

Thanks for your input.

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Tim C. <tc...@op...>]

On Fri, 2002-12-27 at 12:55, Magnus Lie Hetland wrote:
> Tim Churches <tc...@op...>:
> [snip]
> > Have a look at the discussion on RecordArrays in this overview of
> > Numarray: http://stsdas.stsci.edu/numarray/DesignOverview.html
> 
> Sounds interesting.
> 
> > However, in the meantime, as you note, its not too hard to write a class
> > which emulates R/S-Plus data frames. Just store each column in its own
> > Numeric array of the appropriate type
> 
> Yeah -- it's just that I'd like to keep a set of columns collected as
> a two-dimensional array, to allow horizontal summing and the like.
> (Not much more complicated, but an extra issue to address.)
> 
> > (which might be the PyObject
> > types, which can hold any Python object type),
> 
> Hm. Yes. I can't seem to find these anymore. I seem to recall using
> type='o' or something in Numeric, but I can't find the right type
> objects now... (Guess I'm just reading the docs and dir(numeric)
> poorly...) It would be nice if array(['foo']) just worked. Oh, well.

Just like this:

>>> import Numeric
>>> a = Numeric.array(['a','b','c'],typecode=Numeric.PyObject)
>>> a
array([a , b , c ],'O')
>>>

> 
> > By memory-mapping disc-based
> > versions of the  Numeric arrays, and using the BsdDb3 record number
> > database format for the string columns, you can even make a disc-based
> > "record array" which can be larger than available RAM+swap.
> 
> Sounds quite useful, although quite similar to MetaKit. (I suppose I
> could use some functions from numarray on columns in MetaKit... But
> that might just be too weird -- and it would still just be a
> collection of columns :])

I really like MetaKit's column-based storage, but it just doesn't scale
well (on the author's admission, and verified empirically) - beyond a
few 10**5 records, it bogs down terribly, whereas memory-mapped NumPy
plus BsdDb3 recno databse for strings scales well to many tens of
millions of records (or more, but thats as far as I have tested).

Tim C

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Magnus L. H. <ma...@he...>]

Tim Churches <tc...@op...>:
[snip]
> Just like this:
> 
> >>> import Numeric
> >>> a = Numeric.array(['a','b','c'],typecode=Numeric.PyObject)
> >>> a
> array([a , b , c ],'O')
> >>>

As you may have noticed from my previous descriptions, I'm using
numarray, not Numeric. I've used this in Numeric before -- I just
can't find the equivalent functionality in numarray :)

[snip]
> I really like MetaKit's column-based storage,

Me too.

> but it just doesn't scale
> well (on the author's admission, and verified empirically)

Yes, you're right.

> - beyond a
> few 10**5 records, it bogs down terribly, whereas memory-mapped NumPy
> plus BsdDb3 recno databse for strings scales well to many tens of
> millions of records (or more, but thats as far as I have tested).

Impressive!

Now this *does* sound interesting... The project I originally posted
about only has a few hundred records, so I'm only considering numarray
for expressiveness/readability there -- performance is not an issue.
But using bsddb and numarray (or Numeric) together like this seems
useful in many applications.

> Tim C

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Francesc A. <fa...@op...>]

Mensaje citado por: Magnus Lie Hetland <ma...@he...>:

> I'm working on some two-dimensional tables of data, where some data
> are numerical, while other aren't. I'd like to use numarray's
> numerical capabilities with the numerical parts (columns) while
> keeping the data in each row together. (I'm sure this generalizes to
> more dimensions, and to sub-arrays in general, not just rows.)

You may want to have a look at PyTables (http://pytables.sourceforge.net). 
It's designed to be used in scenarios similar to that you are exposing.

It supports Numeric objects and although columns in tables are not 
automatically converted to Numeric o numarray objects, you can build them on 
the flight easily using its powerful selection capabilities.

It uses HDF5 (http://hdf.ncsa.uiuc.edu/HDF5/) format to save its data, so you 
can read PyTables files in a variety of languages and platforms.

Cheers,

Francesc Alted

Re: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Magnus L. H. <ma...@he...>]

Francesc Alted <fa...@op...>:
[snip]
> You may want to have a look at PyTables (http://pytables.sourceforge.net). 
> It's designed to be used in scenarios similar to that you are exposing.
[snip]

Sounds interesting. I'll look into it.

-- 
Magnus Lie Hetland
http://hetland.org