Re: [Pytables-users] Pytables file structure

[Juan M. V. T. <jmv...@gm...>]

Thank you very much Anthony.
Do I have to sign up to store a ticket?

2012/7/15 Anthony Scopatz <sc...@gm...>

> Ahh I see, tricky.
>
> So I think what is killing you is that you are pulling each row of the
> table individually over the network.  Ideally you should be able to do
> something like the following:
>
> f.root.table.cols.my_col[:,n,:]
>
>
> using numpy-esque multidimensional slicing.  However, this fails when I
> just tested it.  So instead, I would just pull over the full column and
> slice using numpy in memory.
>
> my_col = f.root.table.cols.my_col[:]
> my_selection = my_col[:,n,:]
>
>
> We should open a ticket so that the top method works (though I think there
> might already be one).
>
> I hope this helps!
>
> On Sun, Jul 15, 2012 at 4:27 PM, Juan Manuel Vázquez Tovar <
> jmv...@gm...> wrote:
>
>> The column I´m requesting the data from has multidimensional cells, so
>> each time I request data from the table, I need to get a specific row for
>> all the multidimensional cells in the column. I hope this clarifies a bit.
>> I have at the office a Linux workstation, but it is part of a computing
>> cluster where all the users have access, so the files are in a folder of
>> the cluster, not in my hard drive.
>>
>> Thank you,
>> Juanma
>>
>> 2012/7/15 Anthony Scopatz <sc...@gm...>
>>
>>> Rereading the original post, I am a little confused are your trying to
>>> read the whole table, just a couple of rows that meet some condition, or
>>> just one whole column, or one part of the column.
>>>
>>> To request the whole table without looping over each row in Python,
>>> index every element:
>>>
>>> f.root.table[:]
>>>
>>>
>>> To just get certain rows, use where().
>>>
>>> To get a single column, use the cols namespace:
>>>
>>> f.root.table.cols.my_column[:]
>>>
>>>
>>> Why is this file elsewhere on the network?
>>>
>>> Be Well
>>>  Anthony
>>>
>>> On Sun, Jul 15, 2012 at 4:08 PM, Juan Manuel Vázquez Tovar <
>>> jmv...@gm...> wrote:
>>>
>>>> Hello Anthony,
>>>>
>>>> I have to loop over the whole set of rows. Does the where method has
>>>> any advantages in that case?
>>>>
>>>> Thank you,
>>>> Juanma
>>>>
>>>> 2012/7/15 Anthony Scopatz <sc...@gm...>
>>>>
>>>>> Hello Juan,
>>>>>
>>>>> Try using the where() method [1],  It has a lot of nice features under
>>>>> the covers.
>>>>>
>>>>> Be Well
>>>>> Anthony
>>>>>
>>>>> 1.
>>>>> http://pytables.github.com/usersguide/libref.html?highlight=where#tables.Table.where
>>>>>
>>>>> On Sun, Jul 15, 2012 at 4:01 PM, Juan Manuel Vázquez Tovar <
>>>>> jmv...@gm...> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have been using pytables for a few moths. The main structure of my
>>>>>> files has a four column table, two of which have multidimensional cells,
>>>>>> (56,1) and (133,6) respectively. The previous structure had more columns
>>>>>> instead of storing the 56x1 array into the same cell. The largest file has
>>>>>> almost three million rows in the table.
>>>>>> I usually request data from the table looping through the entire
>>>>>> table and getting for each row one specific row of the 133x6 2d array.
>>>>>> Currently, each of the requests can take from 15 sec up to 10
>>>>>> minutes, I believe that depending on the status of the office network.
>>>>>> Could you please advice about how to improve the reading time?
>>>>>> I have tried to compress the data with zlib, but it takes more or
>>>>>> less the same time.
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Juan Manuel
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Live Security Virtual Conference
>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>> Discussions
>>>>>> will include endpoint security, mobile security and the latest in
>>>>>> malware
>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>> _______________________________________________
>>>>>> Pytables-users mailing list
>>>>>> Pyt...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in
>>>>> malware
>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>> _______________________________________________
>>>>> Pytables-users mailing list
>>>>> Pyt...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>