Re: [Pytables-users] append to multiple tables

[Edward V. <edw...@gm...>]

I initially didn't sync at all until after completing writing - about 1
million rows total. My main concern was preventing data corruption. After
seeing the warning I had a sync for every iteration of the inner loop,
which was slow. Syncing after the inner loop is a little slower than not
syncing, but seems fine.
Thanks,
Ed


On Mon, Jun 10, 2013 at 4:37 PM, Francesc Alted <fa...@gm...> wrote:

> Hi Ed,
>
> After fixing the issue, does performance has been enhanced?  I'm the one
> who put the warning, so I'm curious on whether this actually helps
> people or not.
>
> Thanks,
> Francesc
>
> On 6/10/13 3:28 PM, Edward Vogel wrote:
> > Yes, exactly.
> > I'm pulling data out of C that has a 1 to many relationship, and
> > dumping it into pytables for easier analysis. I'm creating extension
> > classes in cython to get access to the C structures.
> > It looks like this (basically, each cv1 has several cv2s):
> >
> > h5.create_table('/', 'cv1', schema_cv1)
> > h5.create_table('/', 'cv2', schema_cv2)
> > cv1_row = h5.root.cv1.row
> > cv2_row = h5.root.cv2.row
> > for cv in sf.itercv():
> >     cv1_row['addr'] = cv['addr']
> >     ...
> >     cv1_row.append()
> >     for cv2 in cv.itercv2():
> >         cv2_row['cv1_addr'] = cv['addr']
> >         cv2_row['foo'] = cv2_row['foo']
> >         ...
> >         cv2_row.append()
> >     h5.root.cv2.flush()  # This fixes issue
> >
> > Adding the flush after the inner loop does fix the issue. (Thanks!)
> > So, my followup question, why do I need a flush after the inner loop,
> > but not when moving from the outer loop to the inner loop?
> >
> > Thanks!
> >
> >
> >
> > On Mon, Jun 10, 2013 at 2:48 PM, Anthony Scopatz <sc...@gm...
> > <mailto:sc...@gm...>> wrote:
> >
> >     Hi Ed,
> >
> >     Are you inside of a nested loop?  You probably just need to flush
> >     after the innermost loop.
> >
> >     Do you have some sample code you can share?
> >
> >     Be Well
> >     Anthony
> >
> >
> >     On Mon, Jun 10, 2013 at 1:44 PM, Edward Vogel
> >     <edw...@gm... <mailto:edw...@gm...>> wrote:
> >
> >         I have a dataset that I want to split between two tables. But,
> >         when I iterate over the data and append to both tables, I get
> >         a warning:
> >
> >         /usr/local/lib/python2.7/site-packages/tables/table.py:2967:
> >         PerformanceWarning: table ``/cv2`` is being preempted from
> >         alive nodes without its buffers being flushed or with some
> >         index being dirty.  This may lead to very ineficient use of
> >         resources and even to fatal errors in certain situations.
> >          Please do a call to the .flush() or .reindex_dirty() methods
> >         on this table before start using other nodes.
> >
> >         However, if I flush after every append, I get awful performance.
> >         Is there a correct way to append to two tables without doing a
> >         flush?
> >         Note, I don't have any indices defined, so it seems
> >         reindex_dirty() doesn't apply.
> >
> >         Thanks,
> >         Ed
> >
> >
> ------------------------------------------------------------------------------
> >         This SF.net email is sponsored by Windows:
> >
> >         Build for Windows Store.
> >
> >         http://p.sf.net/sfu/windows-dev2dev
> >         _______________________________________________
> >         Pytables-users mailing list
> >         Pyt...@li...
> >         <mailto:Pyt...@li...>
> >         https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> >     This SF.net email is sponsored by Windows:
> >
> >     Build for Windows Store.
> >
> >     http://p.sf.net/sfu/windows-dev2dev
> >     _______________________________________________
> >     Pytables-users mailing list
> >     Pyt...@li...
> >     <mailto:Pyt...@li...>
> >     https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > This SF.net email is sponsored by Windows:
> >
> > Build for Windows Store.
> >
> > http://p.sf.net/sfu/windows-dev2dev
> >
> >
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
> --
> Francesc Alted
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>