Re: [Pytables-users] Pytables file reading

[Juan M. V. T. <jmv...@gm...>]

Hi Antonio,

You are right, I don´t need to load the entire table into memory.
The fourth column has multidimensional cells and when I read a single row
from every cell in the column, I almost fill the workstation memory.
I didn´t expect that process to use so much memory, but the fact is that it
uses it.
May be I didn´t explain very well last time.

Thank you,

Juanma

2012/8/5 Antonio Valentino <ant...@ti...>

> Hi Juan Manuel,
>
> Il 04/08/2012 01:55, Juan Manuel Vázquez Tovar ha scritto:
> > Hello all,
> >
> > I´m managing a file close to 26 Gb size. It´s main structure is  a table
> > with a bit more than 8 million rows. The table is made by four columns,
> the
> > first two columns store names, the 3rd one has a 53 items array in each
> > cell and the last column has a 133x6 matrix in each cell.
> > I use to work with a Linux workstation with 24 Gb. My usual way of
> working
> > with the file is to retrieve, from each cell in the 4th column of the
> > table, the same row from the 133x6 matrix.
> > I store the information in a bumpy array with shape 8e6x6. In this
> process
> > I almost use the whole workstation memory.
> > Is there anyway to optimize the memory usage?
>
> I'm not sure to understand.
> My impression is that you do not actually need to have the entire 8e6x6
> matrix in memory at once, is it correct?
>
> In that case you could simply try to load less data using something like
>
> data = table.read(0, 5e7, field='name of the 4-th field')
> process(data)
> data = table.read(5e7, 1e8,  field='name of the 4-th field')
> process(data)
>
> See also [1] and [2].
>
> Does it make sense for you?
>
>
> [1]
> http://pytables.github.com/usersguide/libref.html#table-methods-reading
> [2] http://pytables.github.com/usersguide/libref.html#tables.Table.read
>
> > If not, I have been thinking about splitting the file.
> >
> > Thank you,
> >
> > Juanma
>
>
> cheers
>
> --
> Antonio Valentino
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>