Re: [Pytables-users] Pytables file structure

[Anthony S. <sc...@gm...>]

Hello Juan,

Just make an account at github [1] and then go to the PyTables issues page.

Be Well
Anthony

1. https://github.com/
2. https://github.com/PyTables/PyTables/issues

On Tue, Jul 17, 2012 at 6:27 PM, Juan Manuel Vázquez Tovar <
jmv...@gm...> wrote:

> Thank you very much Anthony.
> Do I have to sign up to store a ticket?
>
>
> 2012/7/15 Anthony Scopatz <sc...@gm...>
>
>> Ahh I see, tricky.
>>
>> So I think what is killing you is that you are pulling each row of the
>> table individually over the network.  Ideally you should be able to do
>> something like the following:
>>
>> f.root.table.cols.my_col[:,n,:]
>>
>>
>> using numpy-esque multidimensional slicing.  However, this fails when I
>> just tested it.  So instead, I would just pull over the full column and
>> slice using numpy in memory.
>>
>> my_col = f.root.table.cols.my_col[:]
>> my_selection = my_col[:,n,:]
>>
>>
>> We should open a ticket so that the top method works (though I think
>> there might already be one).
>>
>> I hope this helps!
>>
>> On Sun, Jul 15, 2012 at 4:27 PM, Juan Manuel Vázquez Tovar <
>> jmv...@gm...> wrote:
>>
>>> The column I´m requesting the data from has multidimensional cells, so
>>> each time I request data from the table, I need to get a specific row for
>>> all the multidimensional cells in the column. I hope this clarifies a bit.
>>> I have at the office a Linux workstation, but it is part of a computing
>>> cluster where all the users have access, so the files are in a folder of
>>> the cluster, not in my hard drive.
>>>
>>> Thank you,
>>> Juanma
>>>
>>> 2012/7/15 Anthony Scopatz <sc...@gm...>
>>>
>>>> Rereading the original post, I am a little confused are your trying to
>>>> read the whole table, just a couple of rows that meet some condition, or
>>>> just one whole column, or one part of the column.
>>>>
>>>> To request the whole table without looping over each row in Python,
>>>> index every element:
>>>>
>>>> f.root.table[:]
>>>>
>>>>
>>>> To just get certain rows, use where().
>>>>
>>>> To get a single column, use the cols namespace:
>>>>
>>>> f.root.table.cols.my_column[:]
>>>>
>>>>
>>>> Why is this file elsewhere on the network?
>>>>
>>>> Be Well
>>>>  Anthony
>>>>
>>>> On Sun, Jul 15, 2012 at 4:08 PM, Juan Manuel Vázquez Tovar <
>>>> jmv...@gm...> wrote:
>>>>
>>>>> Hello Anthony,
>>>>>
>>>>> I have to loop over the whole set of rows. Does the where method has
>>>>> any advantages in that case?
>>>>>
>>>>> Thank you,
>>>>> Juanma
>>>>>
>>>>> 2012/7/15 Anthony Scopatz <sc...@gm...>
>>>>>
>>>>>> Hello Juan,
>>>>>>
>>>>>> Try using the where() method [1],  It has a lot of
>>>>>> nice features under the covers.
>>>>>>
>>>>>> Be Well
>>>>>> Anthony
>>>>>>
>>>>>> 1.
>>>>>> http://pytables.github.com/usersguide/libref.html?highlight=where#tables.Table.where
>>>>>>
>>>>>> On Sun, Jul 15, 2012 at 4:01 PM, Juan Manuel Vázquez Tovar <
>>>>>> jmv...@gm...> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have been using pytables for a few moths. The main structure of my
>>>>>>> files has a four column table, two of which have multidimensional cells,
>>>>>>> (56,1) and (133,6) respectively. The previous structure had more columns
>>>>>>> instead of storing the 56x1 array into the same cell. The largest file has
>>>>>>> almost three million rows in the table.
>>>>>>> I usually request data from the table looping through the entire
>>>>>>> table and getting for each row one specific row of the 133x6 2d array.
>>>>>>> Currently, each of the requests can take from 15 sec up to 10
>>>>>>> minutes, I believe that depending on the status of the office network.
>>>>>>> Could you please advice about how to improve the reading time?
>>>>>>> I have tried to compress the data with zlib, but it takes more or
>>>>>>> less the same time.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> Juan Manuel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>>> Live Security Virtual Conference
>>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>>> Discussions
>>>>>>> will include endpoint security, mobile security and the latest in
>>>>>>> malware
>>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>>> _______________________________________________
>>>>>>> Pytables-users mailing list
>>>>>>> Pyt...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Live Security Virtual Conference
>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>> Discussions
>>>>>> will include endpoint security, mobile security and the latest in
>>>>>> malware
>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>> _______________________________________________
>>>>>> Pytables-users mailing list
>>>>>> Pyt...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in
>>>>> malware
>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>> _______________________________________________
>>>>> Pytables-users mailing list
>>>>> Pyt...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>