Re: [Pytables-users] Pytables file reading

[Juan M. V. T. <jmv...@gm...>]

Thank you Antonio, I will try

Cheers

Juanma

El Aug 5, 2012, a las 17:32, Antonio Valentino <ant...@ti...> escribió:

> Hi Juan Manuel,
> 
> Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto:
>> Hi Antonio,
>> 
>> This is the piece of code I use to read the part of the table I need:
>> 
>> data = [case[´loads´][i] for case in table]
>> 
>> where i is the index of the row that I need to read from the matrix (133x6)
>> stored in each cell of the column "loads".
>> 
>> Juanma
>> 
> 
> that looks perfectly fine to me.
> No idea about what could be the issue :/
> 
> You can perfform patrial reads using Table.iterrows:
> 
> data = [case[´loads´][i] for case in table.iterrows(start, stop)]
> 
> Please also consider that using a single np.array with 1e8 rows instead
> of a list of arrays will allows you to save the memory overhead of 1e8
> array objects.
> Considering that 6 doubles are 48 bytes while an empty np.array takes 80
> bytes
> 
> In [64]: sys.getsizeof(np.zeros((0,)))
> Out[64]: 80
> 
> you should be able to reduce the memory footprint by far more than an half.
> 
> 
> cheers
> 
> 
>> 2012/8/5 Antonio Valentino <ant...@ti...>
>> 
>>> Hi Juan Manuel,
>>> 
>>> Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto:
>>>> Hi Antonio,
>>>> 
>>>> You are right, I don´t need to load the entire table into memory.
>>>> The fourth column has multidimensional cells and when I read a single row
>>>> from every cell in the column, I almost fill the workstation memory.
>>>> I didn´t expect that process to use so much memory, but the fact is that
>>> it
>>>> uses it.
>>>> May be I didn´t explain very well last time.
>>>> 
>>>> Thank you,
>>>> 
>>>> Juanma
>>>> 
>>> 
>>> Sorry, still don't understand.
>>> Can you please post a short code snipped that shows how exactly do you
>>> read data into your program?
>>> 
>>> My impression is that somewhere you use some instruction that triggers
>>> loading of unnecessary data into memory.
> 
> 
> -- 
> Antonio Valentino
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users