Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 2

[David R. <dav...@gm...>]

Thanks Anthony, but unless Im missing something I don't think that method
will work since this will only be comparing the ith element with ith+1
element.  I still need 2 for loops right?

Using itertools might speed things up though, I've never used them so I
will give it a shot and let you know how it goes.  Looks like I need to
download the latest release before I do that too.  Thanks for the help.

-Dave



On Thu, Jan 3, 2013 at 12:12 PM, <
pyt...@li...> wrote:

> Send Pytables-users mailing list submissions to
>         pyt...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.sourceforge.net/lists/listinfo/pytables-users
> or, via email, send a message with subject or body 'help' to
>         pyt...@li...
>
> You can reach the person managing the list at
>         pyt...@li...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pytables-users digest..."
>
>
> Today's Topics:
>
>    1. Re: Nested Iteration of HDF5 using PyTables (Anthony Scopatz)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 3 Jan 2013 11:11:47 -0600
> From: Anthony Scopatz <sc...@gm...>
> Subject: Re: [Pytables-users] Nested Iteration of HDF5 using PyTables
> To: Discussion list for PyTables
>         <pyt...@li...>
> Message-ID:
>         <CAPk-6T5b=
> 1EG...@ma...>
> Content-Type: text/plain; charset="iso-8859-1"
>
> HI David,
>
> Tables and table column iteration have been overhauled fairly recently [1].
>  So you might try creating two iterators, offset by one, and then doing the
> comparison.  I am hacking this out super quick so please forgive me:
>
> from itertools import izip
>
> with tb.openFile(...) as f:
>     data = f.root.data
>     data_i = iter(data)
>     data_j = iter(data)
>     data_i.next() # throw the first value away
>     for i, j in izip(data_i, data_j):
>         compare(i, j)
>
> You get the idea ;)
>
> Be Well
> Anthony
>
> 1. https://github.com/PyTables/PyTables/issues/27
>
>
> On Thu, Jan 3, 2013 at 9:25 AM, David Reed <dav...@gm...> wrote:
>
> > I was hoping someone could help me out here.
> >
> > This is from a post I put up on StackOverflow,
> >
> > I am have a fairly large dataset that I store in HDF5 and access using
> > PyTables. One operation I need to do on this dataset are pairwise
> > comparisons between each of the elements. This requires 2 loops, one to
> > iterate over each element, and an inner loop to iterate over every other
> > element. This operation thus looks at N(N-1)/2 comparisons.
> >
> > For fairly small sets I found it to be faster to dump the contents into a
> > multdimensional numpy array and then do my iteration. I run into problems
> > with large sets because of memory issues and need to access each element
> of
> > the dataset at run time.
> >
> > Putting the elements into an array gives me about 600 comparisons per
> > second, while operating on hdf5 data itself gives me about 300
> comparisons
> > per second.
> >
> > Is there a way to speed this process up?
> >
> > Example follows (this is not my real code, just an example):
> >
> > *Small Set*:
> >
> >
> > with tb.openFile(h5_file, 'r') as f:
> >     data = f.root.data
> >
> >     N_elements = len(data)
> >     elements = np.empty((N_irises, 1e5))
> >
> >     for ii, d in enumerate(data):
> >         elements[ii] = data['element']
> >
> > D = np.empty((N_irises, N_irises))  for ii in xrange(N_elements):
> >     for jj in xrange(ii+1, N_elements):
> >         D[ii, jj] = compare(elements[ii], elements[jj])
> >
> >  *Large Set*:
> >
> >
> > with tb.openFile(h5_file, 'r') as f:
> >     data = f.root.data
> >
> >     N_elements = len(data)
> >
> >     D = np.empty((N_irises, N_irises))
> >     for ii in xrange(N_elements):
> >         for jj in xrange(ii+1, N_elements):
> >              D[ii, jj] = compare(data['element'][ii],
> data['element'][jj])
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> > MVPs and experts. ON SALE this month only -- learn more at:
> > http://p.sf.net/sfu/learnmore_122712
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
>
> ------------------------------------------------------------------------------
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. ON SALE this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122712
>
> ------------------------------
>
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
> End of Pytables-users Digest, Vol 80, Issue 2
> *********************************************
>