Re: [Pytables-users] Pytables file structure

[Anthony S. <sc...@gm...>]

Ahh I see, tricky.

So I think what is killing you is that you are pulling each row of the
table individually over the network.  Ideally you should be able to do
something like the following:

f.root.table.cols.my_col[:,n,:]


using numpy-esque multidimensional slicing.  However, this fails when I
just tested it.  So instead, I would just pull over the full column and
slice using numpy in memory.

my_col = f.root.table.cols.my_col[:]
my_selection = my_col[:,n,:]


We should open a ticket so that the top method works (though I think there
might already be one).

I hope this helps!

On Sun, Jul 15, 2012 at 4:27 PM, Juan Manuel Vázquez Tovar <
jmv...@gm...> wrote:

> The column I´m requesting the data from has multidimensional cells, so
> each time I request data from the table, I need to get a specific row for
> all the multidimensional cells in the column. I hope this clarifies a bit.
> I have at the office a Linux workstation, but it is part of a computing
> cluster where all the users have access, so the files are in a folder of
> the cluster, not in my hard drive.
>
> Thank you,
> Juanma
>
> 2012/7/15 Anthony Scopatz <sc...@gm...>
>
>> Rereading the original post, I am a little confused are your trying to
>> read the whole table, just a couple of rows that meet some condition, or
>> just one whole column, or one part of the column.
>>
>> To request the whole table without looping over each row in Python, index
>> every element:
>>
>> f.root.table[:]
>>
>>
>> To just get certain rows, use where().
>>
>> To get a single column, use the cols namespace:
>>
>> f.root.table.cols.my_column[:]
>>
>>
>> Why is this file elsewhere on the network?
>>
>> Be Well
>>  Anthony
>>
>> On Sun, Jul 15, 2012 at 4:08 PM, Juan Manuel Vázquez Tovar <
>> jmv...@gm...> wrote:
>>
>>> Hello Anthony,
>>>
>>> I have to loop over the whole set of rows. Does the where method has any
>>> advantages in that case?
>>>
>>> Thank you,
>>> Juanma
>>>
>>> 2012/7/15 Anthony Scopatz <sc...@gm...>
>>>
>>>> Hello Juan,
>>>>
>>>> Try using the where() method [1],  It has a lot of nice features under
>>>> the covers.
>>>>
>>>> Be Well
>>>> Anthony
>>>>
>>>> 1.
>>>> http://pytables.github.com/usersguide/libref.html?highlight=where#tables.Table.where
>>>>
>>>> On Sun, Jul 15, 2012 at 4:01 PM, Juan Manuel Vázquez Tovar <
>>>> jmv...@gm...> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have been using pytables for a few moths. The main structure of my
>>>>> files has a four column table, two of which have multidimensional cells,
>>>>> (56,1) and (133,6) respectively. The previous structure had more columns
>>>>> instead of storing the 56x1 array into the same cell. The largest file has
>>>>> almost three million rows in the table.
>>>>> I usually request data from the table looping through the entire table
>>>>> and getting for each row one specific row of the 133x6 2d array.
>>>>> Currently, each of the requests can take from 15 sec up to 10 minutes,
>>>>> I believe that depending on the status of the office network.
>>>>> Could you please advice about how to improve the reading time?
>>>>> I have tried to compress the data with zlib, but it takes more or less
>>>>> the same time.
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Juan Manuel
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in
>>>>> malware
>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>> _______________________________________________
>>>>> Pytables-users mailing list
>>>>> Pyt...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>