Re: [Pytables-users] Pytables file reading

[Antonio V. <ant...@ti...>]

Hi Juan Manuel,

Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto:
> Hi Antonio,
> 
> This is the piece of code I use to read the part of the table I need:
> 
> data = [case[´loads´][i] for case in table]
> 
> where i is the index of the row that I need to read from the matrix (133x6)
> stored in each cell of the column "loads".
> 
> Juanma
> 

that looks perfectly fine to me.
No idea about what could be the issue :/

You can perfform patrial reads using Table.iterrows:

data = [case[´loads´][i] for case in table.iterrows(start, stop)]

Please also consider that using a single np.array with 1e8 rows instead
of a list of arrays will allows you to save the memory overhead of 1e8
array objects.
Considering that 6 doubles are 48 bytes while an empty np.array takes 80
bytes

In [64]: sys.getsizeof(np.zeros((0,)))
Out[64]: 80

you should be able to reduce the memory footprint by far more than an half.


cheers


> 2012/8/5 Antonio Valentino <ant...@ti...>
> 
>> Hi Juan Manuel,
>>
>> Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto:
>>> Hi Antonio,
>>>
>>> You are right, I don´t need to load the entire table into memory.
>>> The fourth column has multidimensional cells and when I read a single row
>>> from every cell in the column, I almost fill the workstation memory.
>>> I didn´t expect that process to use so much memory, but the fact is that
>> it
>>> uses it.
>>> May be I didn´t explain very well last time.
>>>
>>> Thank you,
>>>
>>> Juanma
>>>
>>
>> Sorry, still don't understand.
>> Can you please post a short code snipped that shows how exactly do you
>> read data into your program?
>>
>> My impression is that somewhere you use some instruction that triggers
>> loading of unnecessary data into memory.


-- 
Antonio Valentino