Re: [Pytables-users] Optimizing pytables for reading entire columns at a time

[Anthony S. <sc...@gm...>]

On Fri, Sep 21, 2012 at 10:49 AM, Luke Lee <dur...@gm...> wrote:

> Hi again,
>
> I haven't been getting the updates via email so I'm attempting to post
> again to respond.
>
> Thanks everyone for the suggestions.  I have a few questions:
>
> 1.  What is the benefit of using the stand-alone carray project (
> https://github.com/FrancescAlted/carray) vs Pytables.carray?
>

Hello Luke,

carrays are in-memory, not on disk.


> 2.  I realized my code base never uses the query functionality of a Table.
>  So, I changed all my columns to be just Pytables.carray objects instead.
>  They are all sitting at the top of the hierarchy, just below root.  Is
> this a good idea?
>
> I see a big speed increase from this obviously because now everything is
> stored contiguously.  However, are there any downsides to doing this?  I
> suppose I could also use EArray, but we are never actually changing the
> data once it is stored in HDF5.
>

If it works for you, then great!


> 3.  Is compression automatically happening with the Carray?  I know the
> documentation says that compression is supported, but what do I need to do
> to enable it?  Maybe it's already happening and this is contributing to my
> big speed improvement.
>

For compression to be enabled, you need to define the appropriate filter
[1] on either the node or the file.

4.  I would certainly love to take a look at contributing something like
> this in my free time.  I don't have a whole lot at this time so the changes
> could take a while.  I'm sure I need to learn a lot more about the codebase
> before really giving it a try.  I'm going to take a look at this though,
> thanks for the suggestion!
>

No problem ;)


> 5.  How do I subscribe to the dev mailing list?  I only see announcements
> and users.
>

Here is the dev list site:
https://groups.google.com/forum/?fromgroups#!forum/pytables-dev


> 6.  Any idea why I'm not getting the emails from the list?  I signed up 2
> days ago and didn't get any of your replies via email.
>

We have been having problems with this list.  I think It might be time to
transition...

Be Well
Anthony

1.
http://pytables.github.com/usersguide/libref/helper_classes.html?highlight=filter#tables.Filters


>
>
> ------------------------------------------------------------------------------
> Got visibility?
> Most devs has no idea what their production app looks like.
> Find out how fast your code is with AppDynamics Lite.
> http://ad.doubleclick.net/clk;262219671;13503038;y?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>