Re: [Pytables-users] Balance chunksize in large tables

[Anthony S. <sc...@gm...>]

Hello Nikola,

In general, larger chunk sizes will increase read speed.
 Additionally, your problem sounds like a perfect place to use compression,
which can both decrease storage space and increase read speed (use blosc
compression this).    Please refer to [1] for more information.

In general, if you know *a priori* that you have a hard maximum table size
that you will never go over, you can simply set your chunksize to this
value.  On the other hand, if you know a minimum size that you will be
removing and this size is "large enough" then it makes sense to use this as
the chunksize sometimes too.

Be Well
Anthony

1. http://pytables.github.com/usersguide/optimization.html

On Fri, May 11, 2012 at 5:15 AM, nikola stevanovic <nid...@gm...>wrote:

> *Hi everyone, *
>
> I'm new member and it's nice to meet you all.
> I need some advices about my work with pytables. The problem is next. I'm
> working on some kind of database using pytables and of course hdf5 format.
> I created table with *six columns, row size 92B*. One column in table is
> Time32Col. This column will be *indexed*. Table *will be updated* every
> couple days (rows will be appended on existing table). *Between every
> update users can create queries on table and consume data*. My question
> is how efficiently balance chunksize between updates, because numbers of
> rows in table will be start from *0 to 10 000 000 000* during the time?
> After this number I will start archiving process, i.e. for example remove
> first five billions rows and store in some other table for archiving. Of
> course, I need this balance because *reading speed*. So, what is most
> efficient way for setting chunksize for my problem? Sorry for my english.
>
> *
> Thanks for advice guys.
> Cheers!
> Nikola*
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>