Menu
▾
▴
Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 4
|
From: David R. <dav...@gm...> - 2013-02-01 21:28:17
|
at the error:
result = numpy.empty(shape=nrows, dtype=dtypeField)
nrows = 4620 and dtypeField is ('bool', (17, 9600))
I'm not sure what that means as a dtype, but thats what it is.
Forgive me if I'm being totally naive, but I thought the whole point of
__iter__ with pyttables was to do iteration on the fly, so there is no
preallocation.
If you have any ideas on this I'm all ears.
Thanks again.
Dave
On Fri, Feb 1, 2013 at 3:45 PM, <
pyt...@li...> wrote:
> Send Pytables-users mailing list submissions to
> pyt...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> or, via email, send a message with subject or body 'help' to
> pyt...@li...
>
> You can reach the person managing the list at
> pyt...@li...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pytables-users digest..."
>
>
> Today's Topics:
>
> 1. Re: Pytables-users Digest, Vol 81, Issue 2 (Anthony Scopatz)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 1 Feb 2013 14:44:40 -0600
> From: Anthony Scopatz <sc...@gm...>
> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 2
> To: Discussion list for PyTables
> <pyt...@li...>
> Message-ID:
> <
> CAP...@ma...>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Fri, Feb 1, 2013 at 12:43 PM, David Reed <dav...@gm...>
> wrote:
>
> > Hi Anthony,
> >
> > Thanks for the reply.
> >
> > I honestly don't know how to monitor my Python memory usage, but I'm sure
> > that its caused by out of memory.
> >
>
> Well, I would just run top or process monitor or something while running
> the python script to see what happens to memory usage as the script chugs
> along...
>
>
> > I'm just trying to find out how to fix it. My HDF5 table has 4620 rows
> > and the column I'm iterating over is a 17x9600 boolean matrix. The
> > __iter__ method is preallocating an array that is this size which appears
> > to be root of the error. I was hoping there is a fix somewhere in here
> to
> > not have to do this preallocation.
> >
>
> So a 17x9600 boolean matrix should only be 0.155 MB in space. 4620 of
> these is ~760 MB. If you have 2 GB of memory and you are iterating over 2
> of these (templates & masks) it is conceivable that you are just running
> out of memory. Maybe there is a way that __iter__ could not preallocate
> something that is basically a temporary. What is the dtype of the
> templates array?
>
> Be Well
> Anthony
>
>
> >
> > Thanks again.
> >
> >
> >
> >
> > On Fri, Feb 1, 2013 at 11:12 AM, <
> > pyt...@li...> wrote:
> >
> >> Send Pytables-users mailing list submissions to
> >> pyt...@li...
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> or, via email, send a message with subject or body 'help' to
> >> pyt...@li...
> >>
> >> You can reach the person managing the list at
> >> pyt...@li...
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of Pytables-users digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >> 1. Re: Pytables-users Digest, Vol 80, Issue 9 (Anthony Scopatz)
> >>
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Fri, 1 Feb 2013 10:11:47 -0600
> >> From: Anthony Scopatz <sc...@gm...>
> >> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 9
> >> To: Discussion list for PyTables
> >> <pyt...@li...>
> >> Message-ID:
> >> <
> >> CAP...@ma...>
> >> Content-Type: text/plain; charset="iso-8859-1"
> >>
> >> Hi David,
> >>
> >> Sorry, I haven't had a ton of time recently. You seem to be getting a
> >> memory error on creating a numpy array. This kind of thing typically
> >> happens when you are out of memory. Does this seem to be the case with
> >> you? When this dies, is your memory usage at 100%? If so, this
> algorithm
> >> might require a little tweaking...
> >>
> >> Be Well
> >> Anthony
> >>
> >>
> >> On Fri, Feb 1, 2013 at 6:15 AM, David Reed <dav...@gm...>
> >> wrote:
> >>
> >> > I'm still having problems with this one. I can't tell if this
> something
> >> > dumb Im doing with itertools, or if its something in pytables.
> >> >
> >> > Would appreciate any help.
> >> >
> >> > Thanks
> >> >
> >> >
> >> > On Wed, Jan 30, 2013 at 5:00 PM, David Reed <dav...@gm...
> >> >wrote:
> >> >
> >> >> I think I have to reopen this issue. I have been running fine for
> >> awhile
> >> >> using the combinations method from itertools, but have recently run
> >> into a
> >> >> memory since I have recently quadrupled the size of the hdf file.
> >> >>
> >> >> Here is my code again:
> >> >>
> >> >> from itertools import combinations, izip
> >> >> with tb.openFile(h5_all, 'r') as f:
> >> >> irises = f.root.irises
> >> >>
> >> >> templates = f.root.irises.cols.templates
> >> >> masks = f.root.irises.cols.masks1
> >> >>
> >> >> N_irises = len(irises)
> >> >> index = np.ones((20 * 480), np.bool)
> >> >>
> >> >> print '%i Comparisons' % (N_irises*(N_irises - 1)/2)
> >> >> D = np.empty((N_irises, N_irises))
> >> >> for (t1, m1, ii), (t2, m2, jj) in combinations(izip(templates,
> masks,
> >> >> range(N_irises)), 2):
> >> >> # print ii
> >> >> D[ii, jj] = ham_dist(
> >> >> t1[8, index],
> >> >> t2[:, index],
> >> >> m1[8, index],
> >> >> m2[:, index],
> >> >> )
> >> >>
> >> >> And here is the error:
> >> >>
> >> >> In [10]: get_hd3()
> >> >> 10669890 Comparisons
> >> >>
> >> >>
> >>
> ---------------------------------------------------------------------------
> >> >> MemoryError Traceback (most recent call
> >> >> last)
> >> >> <ipython-input-10-cfb255ce7bd1> in <module>()
> >> >> ----> 1 get_hd3()
> >> >>
> >> >>
> >> >> 118 print '%i Comparisons' % (N_irises*(N_irises
> -
> >> >> 1)/2)
> >> >> 119 D = np.empty((N_irises, N_irises))
> >> >> --> 120 for (t1, m1, ii), (t2, m2, jj) in
> >> >> combinations(izip(temp
> >> >> lates, masks, range(N_irises)), 2):
> >> >> 121 # print ii
> >> >> 122 D[ii, jj] = ham_dist(
> >> >>
> >> >> c:\python27\lib\site-packages\tables\table.pyc in __iter__(self)
> >> >> 3274 for start_row in xrange(0, len(self), nrowsinbuf):
> >> >> 3275 end_row = min([start_row + nrowsinbuf, max_row])
> >> >> -> 3276 buf = table.read(start_row, end_row, 1,
> >> >> field=self.pathname)
> >> >>
> >> >> 3277 for row in buf:
> >> >> 3278 yield row
> >> >>
> >> >> c:\python27\lib\site-packages\tables\table.pyc in read(self, start,
> >> stop,
> >> >> step,
> >> >> field)
> >> >> 1772 (start, stop, step) = self._processRangeRead(start,
> >> stop,
> >> >> step)
> >> >> 1773
> >> >> -> 1774 arr = self._read(start, stop, step, field)
> >> >> 1775 return internal_to_flavor(arr, self.flavor)
> >> >> 1776
> >> >>
> >> >> c:\python27\lib\site-packages\tables\table.pyc in _read(self, start,
> >> >> stop, step,
> >> >> field)
> >> >> 1719 if field:
> >> >> 1720 # Create a container for the results
> >> >> -> 1721 result = numpy.empty(shape=nrows,
> dtype=dtypeField)
> >> >> 1722 else:
> >> >> 1723 # Recarray case
> >> >>
> >> >> MemoryError:
> >> >> > c:\python27\lib\site-packages\tables\table.py(1721)_read()
> >> >> 1720 # Create a container for the results
> >> >> -> 1721 result = numpy.empty(shape=nrows,
> dtype=dtypeField)
> >> >> 1722 else:
> >> >>
> >> >> Also, if you guys see any performance problems in my code, please let
> >> me
> >> >> know.
> >> >>
> >> >> Thank you so much for the help.
> >> >>
> >> >> -Dave
> >> >>
> >> >>
> >> >> On Fri, Jan 4, 2013 at 8:57 AM, <
> >> >> pyt...@li...> wrote:
> >> >>
> >> >>> Send Pytables-users mailing list submissions to
> >> >>> pyt...@li...
> >> >>>
> >> >>> To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> or, via email, send a message with subject or body 'help' to
> >> >>> pyt...@li...
> >> >>>
> >> >>> You can reach the person managing the list at
> >> >>> pyt...@li...
> >> >>>
> >> >>> When replying, please edit your Subject line so it is more specific
> >> >>> than "Re: Contents of Pytables-users digest..."
> >> >>>
> >> >>>
> >> >>> Today's Topics:
> >> >>>
> >> >>> 1. Re: Pytables-users Digest, Vol 80, Issue 8 (David Reed)
> >> >>>
> >> >>>
> >> >>>
> ----------------------------------------------------------------------
> >> >>>
> >> >>> Message: 1
> >> >>> Date: Fri, 4 Jan 2013 08:56:28 -0500
> >> >>> From: David Reed <dav...@gm...>
> >> >>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 8
> >> >>> To: pyt...@li...
> >> >>> Message-ID:
> >> >>> <
> >> >>> CAM...@ma...>
> >> >>> Content-Type: text/plain; charset="iso-8859-1"
> >> >>>
> >> >>> I can't thank you guys enough for the help. I was able to add the
> >> >>> __iter__
> >> >>> function to the table.py file and everything seems to be working
> >> great!
> >> >>> I'm not quite as fast as I was with iterating right of a matrix but
> >> >>> pretty
> >> >>> close. I was at 555 comparisons per second, and now im at 420.
> >> >>>
> >> >>> I handled the problem I mentioned earlier by doing this, and it
> seems
> >> to
> >> >>> work great:
> >> >>>
> >> >>> A = f.root.data.cols.A
> >> >>> B = f.root.data.cols.B
> >> >>>
> >> >>> D = np.empty((len(A), len(A))
> >> >>> for (a1, b1, ii), (a2, b2, jj) in combinations(izip(A, B,
> >> range(len(A))),
> >> >>> 2):
> >> >>> D[ii, jj] = compare(a1, a2, b1, b2)
> >> >>>
> >> >>> Again, thanks a lot.
> >> >>>
> >> >>> -Dave
> >> >>>
> >> >>>
> >> >>> On Thu, Jan 3, 2013 at 6:31 PM, <
> >> >>> pyt...@li...> wrote:
> >> >>>
> >> >>> > Send Pytables-users mailing list submissions to
> >> >>> > pyt...@li...
> >> >>> >
> >> >>> > To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> >
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > or, via email, send a message with subject or body 'help' to
> >> >>> > pyt...@li...
> >> >>> >
> >> >>> > You can reach the person managing the list at
> >> >>> > pyt...@li...
> >> >>> >
> >> >>> > When replying, please edit your Subject line so it is more
> specific
> >> >>> > than "Re: Contents of Pytables-users digest..."
> >> >>> >
> >> >>> >
> >> >>> > Today's Topics:
> >> >>> >
> >> >>> > 1. Re: Pytables-users Digest, Vol 80, Issue 3 (Anthony Scopatz)
> >> >>> > 2. Re: Pytables-users Digest, Vol 80, Issue 4 (Anthony Scopatz)
> >> >>> >
> >> >>> >
> >> >>> >
> >> ----------------------------------------------------------------------
> >> >>> >
> >> >>> > Message: 1
> >> >>> > Date: Thu, 3 Jan 2013 17:26:55 -0600
> >> >>> > From: Anthony Scopatz <sc...@gm...>
> >> >>> > Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80,
> Issue 3
> >> >>> > To: Discussion list for PyTables
> >> >>> > <pyt...@li...>
> >> >>> > Message-ID:
> >> >>> > <CAPk-6T6sz=J5ay_a9YGLPe_yBLGa9c+XgxG0CRNs6fJ=
> >> >>> > Gz...@ma...>
> >> >>> > Content-Type: text/plain; charset="iso-8859-1"
> >> >>> >
> >> >>> > On Thu, Jan 3, 2013 at 2:17 PM, David Reed <
> dav...@gm...>
> >> >>> wrote:
> >> >>> >
> >> >>> > > Thanks a lot for the help so far guys!
> >> >>> > >
> >> >>> > > Looking at itertools, I found what I believe to be the perfect
> >> >>> function
> >> >>> > > for what I need, itertools.combinations. This appears to be a
> >> valid
> >> >>> > > replacement to the method proposed.
> >> >>> > >
> >> >>> >
> >> >>> > Yes, combinations is awesome!
> >> >>> >
> >> >>> >
> >> >>> > >
> >> >>> > > There is a small problem that I didn't mention is that my
> compare
> >> >>> > function
> >> >>> > > actually takes as inputs 2 columns from the table. Like so:
> >> >>> > >
> >> >>> > > D = np.empty((N_irises, N_irises))
> >> >>> > > for ii in xrange(N_elements):
> >> >>> > > for jj in xrange(ii+1, N_elements):
> >> >>> > > D[ii, jj] = compare(data['element1'][ii],
> >> >>> > data['element1'][jj],data['element2'][ii],
> >> >>> > > data['element2'][jj])
> >> >>> > >
> >> >>> > > Is there an efficient way of using itertools with this
> structure?
> >> >>> > >
> >> >>> >
> >> >>> > You can always make two other iterators for each column. Since
> you
> >> >>> have
> >> >>> > two columns you would have 4 iterators. I am not sure how fast
> >> this is
> >> >>> > going to be but I am confident that there is definitely a way to
> do
> >> >>> this in
> >> >>> > one for-loop, which is going to be way faster than nested loops.
> >> >>> >
> >> >>> > Be Well
> >> >>> > Anthony
> >> >>> >
> >> >>> >
> >> >>> > >
> >> >>> > >
> >> >>> > > On Thu, Jan 3, 2013 at 1:29 PM, <
> >> >>> > > pyt...@li...> wrote:
> >> >>> > >
> >> >>> > >> Send Pytables-users mailing list submissions to
> >> >>> > >> pyt...@li...
> >> >>> > >>
> >> >>> > >> To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> > >>
> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >> or, via email, send a message with subject or body 'help' to
> >> >>> > >> pyt...@li...
> >> >>> > >>
> >> >>> > >> You can reach the person managing the list at
> >> >>> > >> pyt...@li...
> >> >>> > >>
> >> >>> > >> When replying, please edit your Subject line so it is more
> >> specific
> >> >>> > >> than "Re: Contents of Pytables-users digest..."
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> Today's Topics:
> >> >>> > >>
> >> >>> > >> 1. Re: Nested Iteration of HDF5 using PyTables (Josh Ayers)
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>
> >> >>>
> ----------------------------------------------------------------------
> >> >>> > >>
> >> >>> > >> Message: 1
> >> >>> > >> Date: Thu, 3 Jan 2013 10:29:33 -0800
> >> >>> > >> From: Josh Ayers <jos...@gm...>
> >> >>> > >> Subject: Re: [Pytables-users] Nested Iteration of HDF5 using
> >> >>> PyTables
> >> >>> > >> To: Discussion list for PyTables
> >> >>> > >> <pyt...@li...>
> >> >>> > >> Message-ID:
> >> >>> > >> <
> >> >>> > >>
> >> CAC...@ma...>
> >> >>> > >> Content-Type: text/plain; charset="iso-8859-1"
> >> >>> > >>
> >> >>> > >> David,
> >> >>> > >>
> >> >>> > >> The change in issue 27 was only for iteration over a
> >> tables.Column
> >> >>> > >> instance. To use it, tweak Anthony's code as follows. This
> will
> >> >>> > iterate
> >> >>> > >> over the "element" column, as in your original example.
> >> >>> > >>
> >> >>> > >> Note also that this will only work with the development version
> >> of
> >> >>> > >> PyTables
> >> >>> > >> available on github. It will be very slow using the released
> >> >>> v2.4.0.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> from itertools import izip
> >> >>> > >>
> >> >>> > >> with tb.openFile(...) as f:
> >> >>> > >> data = f.root.data.cols.element
> >> >>> > >> data_i = iter(data)
> >> >>> > >> data_j = iter(data)
> >> >>> > >> data_i.next() # throw the first value away
> >> >>> > >> for i, j in izip(data_i, data_j):
> >> >>> > >> compare(i, j)
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> Hope that helps,
> >> >>> > >> Josh
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> On Thu, Jan 3, 2013 at 9:11 AM, Anthony Scopatz <
> >> sc...@gm...>
> >> >>> > >> wrote:
> >> >>> > >>
> >> >>> > >> > HI David,
> >> >>> > >> >
> >> >>> > >> > Tables and table column iteration have been overhauled fairly
> >> >>> recently
> >> >>> > >> > [1]. So you might try creating two iterators, offset by one,
> >> and
> >> >>> then
> >> >>> > >> > doing the comparison. I am hacking this out super quick so
> >> please
> >> >>> > >> forgive
> >> >>> > >> > me:
> >> >>> > >> >
> >> >>> > >> > from itertools import izip
> >> >>> > >> >
> >> >>> > >> > with tb.openFile(...) as f:
> >> >>> > >> > data = f.root.data
> >> >>> > >> > data_i = iter(data)
> >> >>> > >> > data_j = iter(data)
> >> >>> > >> > data_i.next() # throw the first value away
> >> >>> > >> > for i, j in izip(data_i, data_j):
> >> >>> > >> > compare(i, j)
> >> >>> > >> >
> >> >>> > >> > You get the idea ;)
> >> >>> > >> >
> >> >>> > >> > Be Well
> >> >>> > >> > Anthony
> >> >>> > >> >
> >> >>> > >> > 1. https://github.com/PyTables/PyTables/issues/27
> >> >>> > >> >
> >> >>> > >> >
> >> >>> > >> > On Thu, Jan 3, 2013 at 9:25 AM, David Reed <
> >> >>> dav...@gm...>
> >> >>> > >> wrote:
> >> >>> > >> >
> >> >>> > >> >> I was hoping someone could help me out here.
> >> >>> > >> >>
> >> >>> > >> >> This is from a post I put up on StackOverflow,
> >> >>> > >> >>
> >> >>> > >> >> I am have a fairly large dataset that I store in HDF5 and
> >> access
> >> >>> > using
> >> >>> > >> >> PyTables. One operation I need to do on this dataset are
> >> pairwise
> >> >>> > >> >> comparisons between each of the elements. This requires 2
> >> loops,
> >> >>> one
> >> >>> > to
> >> >>> > >> >> iterate over each element, and an inner loop to iterate over
> >> >>> every
> >> >>> > >> other
> >> >>> > >> >> element. This operation thus looks at N(N-1)/2 comparisons.
> >> >>> > >> >>
> >> >>> > >> >> For fairly small sets I found it to be faster to dump the
> >> >>> contents
> >> >>> > >> into a
> >> >>> > >> >> multdimensional numpy array and then do my iteration. I run
> >> into
> >> >>> > >> problems
> >> >>> > >> >> with large sets because of memory issues and need to access
> >> each
> >> >>> > >> element of
> >> >>> > >> >> the dataset at run time.
> >> >>> > >> >>
> >> >>> > >> >> Putting the elements into an array gives me about 600
> >> >>> comparisons per
> >> >>> > >> >> second, while operating on hdf5 data itself gives me about
> 300
> >> >>> > >> comparisons
> >> >>> > >> >> per second.
> >> >>> > >> >>
> >> >>> > >> >> Is there a way to speed this process up?
> >> >>> > >> >>
> >> >>> > >> >> Example follows (this is not my real code, just an example):
> >> >>> > >> >>
> >> >>> > >> >> *Small Set*:
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >> >> with tb.openFile(h5_file, 'r') as f:
> >> >>> > >> >> data = f.root.data
> >> >>> > >> >>
> >> >>> > >> >> N_elements = len(data)
> >> >>> > >> >> elements = np.empty((N_irises, 1e5))
> >> >>> > >> >>
> >> >>> > >> >> for ii, d in enumerate(data):
> >> >>> > >> >> elements[ii] = data['element']
> >> >>> > >> >>
> >> >>> > >> >> D = np.empty((N_irises, N_irises)) for ii in
> >> xrange(N_elements):
> >> >>> > >> >> for jj in xrange(ii+1, N_elements):
> >> >>> > >> >> D[ii, jj] = compare(elements[ii], elements[jj])
> >> >>> > >> >>
> >> >>> > >> >> *Large Set*:
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >> >> with tb.openFile(h5_file, 'r') as f:
> >> >>> > >> >> data = f.root.data
> >> >>> > >> >>
> >> >>> > >> >> N_elements = len(data)
> >> >>> > >> >>
> >> >>> > >> >> D = np.empty((N_irises, N_irises))
> >> >>> > >> >> for ii in xrange(N_elements):
> >> >>> > >> >> for jj in xrange(ii+1, N_elements):
> >> >>> > >> >> D[ii, jj] = compare(data['element'][ii],
> >> >>> > >> data['element'][jj])
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >> >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> HTML5,
> >> >>> CSS,
> >> >>> > >> >> MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> skills
> >> >>> > current
> >> >>> > >> >> with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >> >> MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >> >> http://p.sf.net/sfu/learnmore_122712
> >> >>> > >> >> _______________________________________________
> >> >>> > >> >> Pytables-users mailing list
> >> >>> > >> >> Pyt...@li...
> >> >>> > >> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >> >>
> >> >>> > >> >>
> >> >>> > >> >
> >> >>> > >> >
> >> >>> > >> >
> >> >>> > >>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> HTML5,
> >> >>> CSS,
> >> >>> > >> > MVC, Windows 8 Apps, JavaScript and much more. Keep your
> skills
> >> >>> > current
> >> >>> > >> > with LearnDevNow - 3,200 step-by-step video tutorials by
> >> Microsoft
> >> >>> > >> > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >> > http://p.sf.net/sfu/learnmore_122712
> >> >>> > >> > _______________________________________________
> >> >>> > >> > Pytables-users mailing list
> >> >>> > >> > Pyt...@li...
> >> >>> > >> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >> >
> >> >>> > >> >
> >> >>> > >> -------------- next part --------------
> >> >>> > >> An HTML attachment was scrubbed...
> >> >>> > >>
> >> >>> > >> ------------------------------
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> HTML5,
> >> >>> CSS,
> >> >>> > >> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> >>> current
> >> >>> > >> with LearnDevNow - 3,200 step-by-step video tutorials by
> >> Microsoft
> >> >>> > >> MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >> http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>
> >> >>> > >> ------------------------------
> >> >>> > >>
> >> >>> > >> _______________________________________________
> >> >>> > >> Pytables-users mailing list
> >> >>> > >> Pyt...@li...
> >> >>> > >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> End of Pytables-users Digest, Vol 80, Issue 3
> >> >>> > >> *********************************************
> >> >>> > >>
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5,
> >> CSS,
> >> >>> > > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> >>> current
> >> >>> > > with LearnDevNow - 3,200 step-by-step video tutorials by
> Microsoft
> >> >>> > > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > > http://p.sf.net/sfu/learnmore_122712
> >> >>> > > _______________________________________________
> >> >>> > > Pytables-users mailing list
> >> >>> > > Pyt...@li...
> >> >>> > > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >
> >> >>> > >
> >> >>> > -------------- next part --------------
> >> >>> > An HTML attachment was scrubbed...
> >> >>> >
> >> >>> > ------------------------------
> >> >>> >
> >> >>> > Message: 2
> >> >>> > Date: Thu, 3 Jan 2013 17:30:59 -0600
> >> >>> > From: Anthony Scopatz <sc...@gm...>
> >> >>> > Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80,
> Issue 4
> >> >>> > To: Discussion list for PyTables
> >> >>> > <pyt...@li...>
> >> >>> > Message-ID:
> >> >>> > <
> >> >>> >
> CAP...@ma...>
> >> >>> > Content-Type: text/plain; charset="iso-8859-1"
> >> >>> >
> >> >>> > Josh is right that you can just edit the code by hand (which works
> >> but
> >> >>> > sucks).
> >> >>> >
> >> >>> > However, on Windows -- on the rare occasion when I also have to
> >> >>> develop on
> >> >>> > it -- I typically use a distribution that includes a compiler,
> >> cython,
> >> >>> > hdf5, and pytables already and then I install my development
> version
> >> >>> from
> >> >>> > github OVER this. I recommend either EPD or Anaconda, though
> other
> >> >>> > distributions listed here [1] might also work.
> >> >>> >
> >> >>> > Be well
> >> >>> > Anthony
> >> >>> >
> >> >>> > 1. http://numfocus.org/projects-2/software-distributions/
> >> >>> >
> >> >>> >
> >> >>> > On Thu, Jan 3, 2013 at 3:46 PM, Josh Ayers <jos...@gm...>
> >> >>> wrote:
> >> >>> >
> >> >>> > > The change was in pure Python code, so you should be able to
> just
> >> >>> paste
> >> >>> > in
> >> >>> > > the changes to your local copy. Start with the
> >> table.Column.__iter__
> >> >>> > > method (lines 3296-3310) here.
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> https://github.com/PyTables/PyTables/blob/b479ed025f4636f7f4744ac83a89bc947808907c/tables/table.py
> >> >>> > >
> >> >>> > > It needs to be modified slightly because it uses some additional
> >> >>> features
> >> >>> > > that aren't available in the released version (the out=buf_slice
> >> >>> argument
> >> >>> > > to table.read). The following should work.
> >> >>> > >
> >> >>> > > def __iter__(self):
> >> >>> > > table = self.table
> >> >>> > > itemsize = self.dtype.itemsize
> >> >>> > > nrowsinbuf = table._v_file.params['IO_BUFFER_SIZE'] //
> >> >>> itemsize
> >> >>> > > max_row = len(self)
> >> >>> > > for start_row in xrange(0, len(self), nrowsinbuf):
> >> >>> > > end_row = min([start_row + nrowsinbuf, max_row])
> >> >>> > > buf = table.read(start_row, end_row, 1,
> >> >>> field=self.pathname)
> >> >>> > > for row in buf:
> >> >>> > > yield row
> >> >>> > >
> >> >>> > >
> >> >>> > > I haven't tested this, but I think it will work.
> >> >>> > >
> >> >>> > > Josh
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> > > On Thu, Jan 3, 2013 at 1:25 PM, David Reed <
> >> dav...@gm...>
> >> >>> > wrote:
> >> >>> > >
> >> >>> > >> I apologize if I'm starting to sound helpless, but I'm forced
> to
> >> >>> work on
> >> >>> > >> Windows 7 at work and have never had luck compiling python
> source
> >> >>> > >> successfully. I have had to rely on precompiled binaries and
> now
> >> >>> its
> >> >>> > >> biting me in the butt.
> >> >>> > >>
> >> >>> > >> Is there any quick fix I can do to improve this iteration using
> >> >>> v2.4.0?
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> On Thu, Jan 3, 2013 at 3:17 PM, <
> >> >>> > >> pyt...@li...> wrote:
> >> >>> > >>
> >> >>> > >>> Send Pytables-users mailing list submissions to
> >> >>> > >>> pyt...@li...
> >> >>> > >>>
> >> >>> > >>> To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> > >>>
> >> >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> or, via email, send a message with subject or body 'help' to
> >> >>> > >>> pyt...@li...
> >> >>> > >>>
> >> >>> > >>> You can reach the person managing the list at
> >> >>> > >>> pyt...@li...
> >> >>> > >>>
> >> >>> > >>> When replying, please edit your Subject line so it is more
> >> specific
> >> >>> > >>> than "Re: Contents of Pytables-users digest..."
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>> Today's Topics:
> >> >>> > >>>
> >> >>> > >>> 1. Re: Pytables-users Digest, Vol 80, Issue 2 (David Reed)
> >> >>> > >>> 2. Re: Pytables-users Digest, Vol 80, Issue 3 (David Reed)
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>>
> >> >>>
> ----------------------------------------------------------------------
> >> >>> > >>>
> >> >>> > >>> Message: 1
> >> >>> > >>> Date: Thu, 3 Jan 2013 13:44:29 -0500
> >> >>> > >>> From: David Reed <dav...@gm...>
> >> >>> > >>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80,
> >> Issue
> >> >>> 2
> >> >>> > >>> To: pyt...@li...
> >> >>> > >>> Message-ID:
> >> >>> > >>> <CAM6XA7=8ocg5WPD4KLSvLhSw-3BCvq5u7MRxq3Ajd6ha=
> >> >>> > >>> ev...@ma...>
> >> >>> > >>> Content-Type: text/plain; charset="iso-8859-1"
> >> >>> > >>>
> >> >>> > >>> Thanks Anthony, but unless Im missing something I don't think
> >> that
> >> >>> > method
> >> >>> > >>> will work since this will only be comparing the ith element
> with
> >> >>> ith+1
> >> >>> > >>> element. I still need 2 for loops right?
> >> >>> > >>>
> >> >>> > >>> Using itertools might speed things up though, I've never used
> >> them
> >> >>> so I
> >> >>> > >>> will give it a shot and let you know how it goes. Looks like
> I
> >> >>> need to
> >> >>> > >>> download the latest release before I do that too. Thanks for
> >> the
> >> >>> help.
> >> >>> > >>>
> >> >>> > >>> -Dave
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>> On Thu, Jan 3, 2013 at 12:12 PM, <
> >> >>> > >>> pyt...@li...> wrote:
> >> >>> > >>>
> >> >>> > >>> > Send Pytables-users mailing list submissions to
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> > >>> >
> >> >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> > or, via email, send a message with subject or body 'help' to
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > You can reach the person managing the list at
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > When replying, please edit your Subject line so it is more
> >> >>> specific
> >> >>> > >>> > than "Re: Contents of Pytables-users digest..."
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > Today's Topics:
> >> >>> > >>> >
> >> >>> > >>> > 1. Re: Nested Iteration of HDF5 using PyTables (Anthony
> >> >>> Scopatz)
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> >
> >> ----------------------------------------------------------------------
> >> >>> > >>> >
> >> >>> > >>> > Message: 1
> >> >>> > >>> > Date: Thu, 3 Jan 2013 11:11:47 -0600
> >> >>> > >>> > From: Anthony Scopatz <sc...@gm...>
> >> >>> > >>> > Subject: Re: [Pytables-users] Nested Iteration of HDF5 using
> >> >>> PyTables
> >> >>> > >>> > To: Discussion list for PyTables
> >> >>> > >>> > <pyt...@li...>
> >> >>> > >>> > Message-ID:
> >> >>> > >>> > <CAPk-6T5b=
> >> >>> > >>> > 1EG...@ma...>
> >> >>> > >>> > Content-Type: text/plain; charset="iso-8859-1"
> >> >>> > >>> >
> >> >>> > >>> > HI David,
> >> >>> > >>> >
> >> >>> > >>> > Tables and table column iteration have been overhauled
> fairly
> >> >>> > recently
> >> >>> > >>> [1].
> >> >>> > >>> > So you might try creating two iterators, offset by one, and
> >> then
> >> >>> > >>> doing the
> >> >>> > >>> > comparison. I am hacking this out super quick so please
> >> forgive
> >> >>> me:
> >> >>> > >>> >
> >> >>> > >>> > from itertools import izip
> >> >>> > >>> >
> >> >>> > >>> > with tb.openFile(...) as f:
> >> >>> > >>> > data = f.root.data
> >> >>> > >>> > data_i = iter(data)
> >> >>> > >>> > data_j = iter(data)
> >> >>> > >>> > data_i.next() # throw the first value away
> >> >>> > >>> > for i, j in izip(data_i, data_j):
> >> >>> > >>> > compare(i, j)
> >> >>> > >>> >
> >> >>> > >>> > You get the idea ;)
> >> >>> > >>> >
> >> >>> > >>> > Be Well
> >> >>> > >>> > Anthony
> >> >>> > >>> >
> >> >>> > >>> > 1. https://github.com/PyTables/PyTables/issues/27
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > On Thu, Jan 3, 2013 at 9:25 AM, David Reed <
> >> >>> dav...@gm...>
> >> >>> > >>> wrote:
> >> >>> > >>> >
> >> >>> > >>> > > I was hoping someone could help me out here.
> >> >>> > >>> > >
> >> >>> > >>> > > This is from a post I put up on StackOverflow,
> >> >>> > >>> > >
> >> >>> > >>> > > I am have a fairly large dataset that I store in HDF5 and
> >> >>> access
> >> >>> > >>> using
> >> >>> > >>> > > PyTables. One operation I need to do on this dataset are
> >> >>> pairwise
> >> >>> > >>> > > comparisons between each of the elements. This requires 2
> >> >>> loops,
> >> >>> > one
> >> >>> > >>> to
> >> >>> > >>> > > iterate over each element, and an inner loop to iterate
> over
> >> >>> every
> >> >>> > >>> other
> >> >>> > >>> > > element. This operation thus looks at N(N-1)/2
> comparisons.
> >> >>> > >>> > >
> >> >>> > >>> > > For fairly small sets I found it to be faster to dump the
> >> >>> contents
> >> >>> > >>> into a
> >> >>> > >>> > > multdimensional numpy array and then do my iteration. I
> run
> >> >>> into
> >> >>> > >>> problems
> >> >>> > >>> > > with large sets because of memory issues and need to
> access
> >> >>> each
> >> >>> > >>> element
> >> >>> > >>> > of
> >> >>> > >>> > > the dataset at run time.
> >> >>> > >>> > >
> >> >>> > >>> > > Putting the elements into an array gives me about 600
> >> >>> comparisons
> >> >>> > per
> >> >>> > >>> > > second, while operating on hdf5 data itself gives me about
> >> 300
> >> >>> > >>> > comparisons
> >> >>> > >>> > > per second.
> >> >>> > >>> > >
> >> >>> > >>> > > Is there a way to speed this process up?
> >> >>> > >>> > >
> >> >>> > >>> > > Example follows (this is not my real code, just an
> example):
> >> >>> > >>> > >
> >> >>> > >>> > > *Small Set*:
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > > with tb.openFile(h5_file, 'r') as f:
> >> >>> > >>> > > data = f.root.data
> >> >>> > >>> > >
> >> >>> > >>> > > N_elements = len(data)
> >> >>> > >>> > > elements = np.empty((N_irises, 1e5))
> >> >>> > >>> > >
> >> >>> > >>> > > for ii, d in enumerate(data):
> >> >>> > >>> > > elements[ii] = data['element']
> >> >>> > >>> > >
> >> >>> > >>> > > D = np.empty((N_irises, N_irises)) for ii in
> >> >>> xrange(N_elements):
> >> >>> > >>> > > for jj in xrange(ii+1, N_elements):
> >> >>> > >>> > > D[ii, jj] = compare(elements[ii], elements[jj])
> >> >>> > >>> > >
> >> >>> > >>> > > *Large Set*:
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > > with tb.openFile(h5_file, 'r') as f:
> >> >>> > >>> > > data = f.root.data
> >> >>> > >>> > >
> >> >>> > >>> > > N_elements = len(data)
> >> >>> > >>> > >
> >> >>> > >>> > > D = np.empty((N_irises, N_irises))
> >> >>> > >>> > > for ii in xrange(N_elements):
> >> >>> > >>> > > for jj in xrange(ii+1, N_elements):
> >> >>> > >>> > > D[ii, jj] = compare(data['element'][ii],
> >> >>> > >>> > data['element'][jj])
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> >
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> > > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> >>> HTML5,
> >> >>> > CSS,
> >> >>> > >>> > > MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> skills
> >> >>> > >>> current
> >> >>> > >>> > > with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >>> > > MVPs and experts. ON SALE this month only -- learn more
> at:
> >> >>> > >>> > > http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>> > > _______________________________________________
> >> >>> > >>> > > Pytables-users mailing list
> >> >>> > >>> > > Pyt...@li...
> >> >>> > >>> > >
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > -------------- next part --------------
> >> >>> > >>> > An HTML attachment was scrubbed...
> >> >>> > >>> >
> >> >>> > >>> > ------------------------------
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> HTML5,
> >> >>> CSS,
> >> >>> > >>> > MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> skills
> >> >>> > current
> >> >>> > >>> > with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >>> > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >>> > http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>> >
> >> >>> > >>> > ------------------------------
> >> >>> > >>> >
> >> >>> > >>> > _______________________________________________
> >> >>> > >>> > Pytables-users mailing list
> >> >>> > >>> > Pyt...@li...
> >> >>> > >>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > End of Pytables-users Digest, Vol 80, Issue 2
> >> >>> > >>> > *********************************************
> >> >>> > >>> >
> >> >>> > >>> -------------- next part --------------
> >> >>> > >>> An HTML attachment was scrubbed...
> >> >>> > >>>
> >> >>> > >>> ------------------------------
> >> >>> > >>>
> >> >>> > >>> Message: 2
> >> >>> > >>> Date: Thu, 3 Jan 2013 15:17:01 -0500
> >> >>> > >>> From: David Reed <dav...@gm...>
> >> >>> > >>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80,
> >> Issue
> >> >>> 3
> >> >>> > >>> To: pyt...@li...
> >> >>> > >>> Message-ID:
> >> >>> > >>> <
> >> >>> > >>>
> >> CAM...@ma...
> >> >>> >
> >> >>> > >>> Content-Type: text/plain; charset="iso-8859-1"
> >> >>> > >>>
> >> >>> > >>> Thanks a lot for the help so far guys!
> >> >>> > >>>
> >> >>> > >>> Looking at itertools, I found what I believe to be the perfect
> >> >>> function
> >> >>> > >>> for
> >> >>> > >>> what I need, itertools.combinations. This appears to be a
> valid
> >> >>> > >>> replacement
> >> >>> > >>> to the method proposed.
> >> >>> > >>>
> >> >>> > >>> There is a small problem that I didn't mention is that my
> >> compare
> >> >>> > >>> function
> >> >>> > >>> actually takes as inputs 2 columns from the table. Like so:
> >> >>> > >>>
> >> >>> > >>> D = np.empty((N_irises, N_irises))
> >> >>> > >>> for ii in xrange(N_elements):
> >> >>> > >>> for jj in xrange(ii+1, N_elements):
> >> >>> > >>> D[ii, jj] = compare(data['element1'][ii],
> >> >>> > >>> data['element1'][jj],data['element2'][ii],
> >> >>> > >>> data['element2'][jj])
> >> >>> > >>>
> >> >>> > >>> Is there an efficient way of using itertools with this
> >> structure?
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>> On Thu, Jan 3, 2013 at 1:29 PM, <
> >> >>> > >>> pyt...@li...> wrote:
> >> >>> > >>>
> >> >>> > >>> > Send Pytables-users mailing list submissions to
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > To subscribe or unsubscribe via the World Wide Web, visit
> >> >>> > >>> >
> >> >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> > or, via email, send a message with subject or body 'help' to
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > You can reach the person managing the list at
> >> >>> > >>> > pyt...@li...
> >> >>> > >>> >
> >> >>> > >>> > When replying, please edit your Subject line so it is more
> >> >>> specific
> >> >>> > >>> > than "Re: Contents of Pytables-users digest..."
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > Today's Topics:
> >> >>> > >>> >
> >> >>> > >>> > 1. Re: Nested Iteration of HDF5 using PyTables (Josh
> Ayers)
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> >
> >> ----------------------------------------------------------------------
> >> >>> > >>> >
> >> >>> > >>> > Message: 1
> >> >>> > >>> > Date: Thu, 3 Jan 2013 10:29:33 -0800
> >> >>> > >>> > From: Josh Ayers <jos...@gm...>
> >> >>> > >>> > Subject: Re: [Pytables-users] Nested Iteration of HDF5 using
> >> >>> PyTables
> >> >>> > >>> > To: Discussion list for PyTables
> >> >>> > >>> > <pyt...@li...>
> >> >>> > >>> > Message-ID:
> >> >>> > >>> > <
> >> >>> > >>> >
> >> >>> CAC...@ma...>
> >> >>> > >>> > Content-Type: text/plain; charset="iso-8859-1"
> >> >>> > >>> >
> >> >>> > >>> > David,
> >> >>> > >>> >
> >> >>> > >>> > The change in issue 27 was only for iteration over a
> >> >>> tables.Column
> >> >>> > >>> > instance. To use it, tweak Anthony's code as follows. This
> >> will
> >> >>> > >>> iterate
> >> >>> > >>> > over the "element" column, as in your original example.
> >> >>> > >>> >
> >> >>> > >>> > Note also that this will only work with the development
> >> version
> >> >>> of
> >> >>> > >>> PyTables
> >> >>> > >>> > available on github. It will be very slow using the
> released
> >> >>> v2.4.0.
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > from itertools import izip
> >> >>> > >>> >
> >> >>> > >>> > with tb.openFile(...) as f:
> >> >>> > >>> > data = f.root.data.cols.element
> >> >>> > >>> > data_i = iter(data)
> >> >>> > >>> > data_j = iter(data)
> >> >>> > >>> > data_i.next() # throw the first value away
> >> >>> > >>> > for i, j in izip(data_i, data_j):
> >> >>> > >>> > compare(i, j)
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > Hope that helps,
> >> >>> > >>> > Josh
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > On Thu, Jan 3, 2013 at 9:11 AM, Anthony Scopatz <
> >> >>> sc...@gm...>
> >> >>> > >>> wrote:
> >> >>> > >>> >
> >> >>> > >>> > > HI David,
> >> >>> > >>> > >
> >> >>> > >>> > > Tables and table column iteration have been overhauled
> >> fairly
> >> >>> > >>> recently
> >> >>> > >>> > > [1]. So you might try creating two iterators, offset by
> >> one,
> >> >>> and
> >> >>> > >>> then
> >> >>> > >>> > > doing the comparison. I am hacking this out super quick
> so
> >> >>> please
> >> >>> > >>> > forgive
> >> >>> > >>> > > me:
> >> >>> > >>> > >
> >> >>> > >>> > > from itertools import izip
> >> >>> > >>> > >
> >> >>> > >>> > > with tb.openFile(...) as f:
> >> >>> > >>> > > data = f.root.data
> >> >>> > >>> > > data_i = iter(data)
> >> >>> > >>> > > data_j = iter(data)
> >> >>> > >>> > > data_i.next() # throw the first value away
> >> >>> > >>> > > for i, j in izip(data_i, data_j):
> >> >>> > >>> > > compare(i, j)
> >> >>> > >>> > >
> >> >>> > >>> > > You get the idea ;)
> >> >>> > >>> > >
> >> >>> > >>> > > Be Well
> >> >>> > >>> > > Anthony
> >> >>> > >>> > >
> >> >>> > >>> > > 1. https://github.com/PyTables/PyTables/issues/27
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > > On Thu, Jan 3, 2013 at 9:25 AM, David Reed <
> >> >>> dav...@gm...
> >> >>> > >
> >> >>> > >>> > wrote:
> >> >>> > >>> > >
> >> >>> > >>> > >> I was hoping someone could help me out here.
> >> >>> > >>> > >>
> >> >>> > >>> > >> This is from a post I put up on StackOverflow,
> >> >>> > >>> > >>
> >> >>> > >>> > >> I am have a fairly large dataset that I store in HDF5 and
> >> >>> access
> >> >>> > >>> using
> >> >>> > >>> > >> PyTables. One operation I need to do on this dataset are
> >> >>> pairwise
> >> >>> > >>> > >> comparisons between each of the elements. This requires 2
> >> >>> loops,
> >> >>> > >>> one to
> >> >>> > >>> > >> iterate over each element, and an inner loop to iterate
> >> over
> >> >>> every
> >> >>> > >>> other
> >> >>> > >>> > >> element. This operation thus looks at N(N-1)/2
> comparisons.
> >> >>> > >>> > >>
> >> >>> > >>> > >> For fairly small sets I found it to be faster to dump the
> >> >>> contents
> >> >>> > >>> into
> >> >>> > >>> > a
> >> >>> > >>> > >> multdimensional numpy array and then do my iteration. I
> run
> >> >>> into
> >> >>> > >>> > problems
> >> >>> > >>> > >> with large sets because of memory issues and need to
> access
> >> >>> each
> >> >>> > >>> > element of
> >> >>> > >>> > >> the dataset at run time.
> >> >>> > >>> > >>
> >> >>> > >>> > >> Putting the elements into an array gives me about 600
> >> >>> comparisons
> >> >>> > >>> per
> >> >>> > >>> > >> second, while operating on hdf5 data itself gives me
> about
> >> 300
> >> >>> > >>> > comparisons
> >> >>> > >>> > >> per second.
> >> >>> > >>> > >>
> >> >>> > >>> > >> Is there a way to speed this process up?
> >> >>> > >>> > >>
> >> >>> > >>> > >> Example follows (this is not my real code, just an
> >> example):
> >> >>> > >>> > >>
> >> >>> > >>> > >> *Small Set*:
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> > >> with tb.openFile(h5_file, 'r') as f:
> >> >>> > >>> > >> data = f.root.data
> >> >>> > >>> > >>
> >> >>> > >>> > >> N_elements = len(data)
> >> >>> > >>> > >> elements = np.empty((N_irises, 1e5))
> >> >>> > >>> > >>
> >> >>> > >>> > >> for ii, d in enumerate(data):
> >> >>> > >>> > >> elements[ii] = data['element']
> >> >>> > >>> > >>
> >> >>> > >>> > >> D = np.empty((N_irises, N_irises)) for ii in
> >> >>> xrange(N_elements):
> >> >>> > >>> > >> for jj in xrange(ii+1, N_elements):
> >> >>> > >>> > >> D[ii, jj] = compare(elements[ii], elements[jj])
> >> >>> > >>> > >>
> >> >>> > >>> > >> *Large Set*:
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> > >> with tb.openFile(h5_file, 'r') as f:
> >> >>> > >>> > >> data = f.root.data
> >> >>> > >>> > >>
> >> >>> > >>> > >> N_elements = len(data)
> >> >>> > >>> > >>
> >> >>> > >>> > >> D = np.empty((N_irises, N_irises))
> >> >>> > >>> > >> for ii in xrange(N_elements):
> >> >>> > >>> > >> for jj in xrange(ii+1, N_elements):
> >> >>> > >>> > >> D[ii, jj] = compare(data['element'][ii],
> >> >>> > >>> > data['element'][jj])
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> >
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> > >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> >>> HTML5,
> >> >>> > >>> CSS,
> >> >>> > >>> > >> MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> >>> skills
> >> >>> > >>> current
> >> >>> > >>> > >> with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >>> > >> MVPs and experts. ON SALE this month only -- learn more
> at:
> >> >>> > >>> > >> http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>> > >> _______________________________________________
> >> >>> > >>> > >> Pytables-users mailing list
> >> >>> > >>> > >> Pyt...@li...
> >> >>> > >>> > >>
> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> > >>
> >> >>> > >>> > >>
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> >
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> > > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> >>> HTML5,
> >> >>> > CSS,
> >> >>> > >>> > > MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> skills
> >> >>> > >>> current
> >> >>> > >>> > > with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >>> > > MVPs and experts. ON SALE this month only -- learn more
> at:
> >> >>> > >>> > > http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>> > > _______________________________________________
> >> >>> > >>> > > Pytables-users mailing list
> >> >>> > >>> > > Pyt...@li...
> >> >>> > >>> > >
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> > >
> >> >>> > >>> > >
> >> >>> > >>> > -------------- next part --------------
> >> >>> > >>> > An HTML attachment was scrubbed...
> >> >>> > >>> >
> >> >>> > >>> > ------------------------------
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> >> HTML5,
> >> >>> CSS,
> >> >>> > >>> > MVC, Windows 8 Apps, JavaScript and much more. Keep your
> >> skills
> >> >>> > current
> >> >>> > >>> > with LearnDevNow - 3,200 step-by-step video tutorials by
> >> >>> Microsoft
> >> >>> > >>> > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >>> > http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>> >
> >> >>> > >>> > ------------------------------
> >> >>> > >>> >
> >> >>> > >>> > _______________________________________________
> >> >>> > >>> > Pytables-users mailing list
> >> >>> > >>> > Pyt...@li...
> >> >>> > >>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>> >
> >> >>> > >>> >
> >> >>> > >>> > End of Pytables-users Digest, Vol 80, Issue 3
> >> >>> > >>> > *********************************************
> >> >>> > >>> >
> >> >>> > >>> -------------- next part --------------
> >> >>> > >>> An HTML attachment was scrubbed...
> >> >>> > >>>
> >> >>> > >>> ------------------------------
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >>> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> HTML5,
> >> >>> CSS,
> >> >>> > >>> MVC, Windows 8 Apps, JavaScript and much more. Keep your
> skills
> >> >>> current
> >> >>> > >>> with LearnDevNow - 3,200 step-by-step video tutorials by
> >> Microsoft
> >> >>> > >>> MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >>> http://p.sf.net/sfu/learnmore_122712
> >> >>> > >>>
> >> >>> > >>> ------------------------------
> >> >>> > >>>
> >> >>> > >>> _______________________________________________
> >> >>> > >>> Pytables-users mailing list
> >> >>> > >>> Pyt...@li...
> >> >>> > >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>> End of Pytables-users Digest, Vol 80, Issue 4
> >> >>> > >>> *********************************************
> >> >>> > >>>
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
> HTML5,
> >> >>> CSS,
> >> >>> > >> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> >>> current
> >> >>> > >> with LearnDevNow - 3,200 step-by-step video tutorials by
> >> Microsoft
> >> >>> > >> MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > >> http://p.sf.net/sfu/learnmore_122712
> >> >>> > >> _______________________________________________
> >> >>> > >> Pytables-users mailing list
> >> >>> > >> Pyt...@li...
> >> >>> > >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >>
> >> >>> > >>
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5,
> >> CSS,
> >> >>> > > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> >>> current
> >> >>> > > with LearnDevNow - 3,200 step-by-step video tutorials by
> Microsoft
> >> >>> > > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > > http://p.sf.net/sfu/learnmore_122712
> >> >>> > > _______________________________________________
> >> >>> > > Pytables-users mailing list
> >> >>> > > Pyt...@li...
> >> >>> > > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> > >
> >> >>> > >
> >> >>> > -------------- next part --------------
> >> >>> > An HTML attachment was scrubbed...
> >> >>> >
> >> >>> > ------------------------------
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5,
> >> CSS,
> >> >>> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> current
> >> >>> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> >> >>> > MVPs and experts. ON SALE this month only -- learn more at:
> >> >>> > http://p.sf.net/sfu/learnmore_122712
> >> >>> >
> >> >>> > ------------------------------
> >> >>> >
> >> >>> > _______________________________________________
> >> >>> > Pytables-users mailing list
> >> >>> > Pyt...@li...
> >> >>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>> >
> >> >>> >
> >> >>> > End of Pytables-users Digest, Vol 80, Issue 8
> >> >>> > *********************************************
> >> >>> >
> >> >>> -------------- next part --------------
> >> >>> An HTML attachment was scrubbed...
> >> >>>
> >> >>> ------------------------------
> >> >>>
> >> >>>
> >> >>>
> >>
> ------------------------------------------------------------------------------
> >> >>> Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and
> >> >>> much more. Get web development skills now with LearnDevNow -
> >> >>> 350+ hours of step-by-step video tutorials by Microsoft MVPs and
> >> experts.
> >> >>> SALE $99.99 this month only -- learn more at:
> >> >>> http://p.sf.net/sfu/learnmore_122812
> >> >>>
> >> >>> ------------------------------
> >> >>>
> >> >>> _______________________________________________
> >> >>> Pytables-users mailing list
> >> >>> Pyt...@li...
> >> >>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >>>
> >> >>>
> >> >>> End of Pytables-users Digest, Vol 80, Issue 9
> >> >>> *********************************************
> >> >>>
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >>
> ------------------------------------------------------------------------------
> >> > Everyone hates slow websites. So do we.
> >> > Make your web apps faster with AppDynamics
> >> > Download AppDynamics Lite for free today:
> >> > http://p.sf.net/sfu/appdyn_d2d_jan
> >> > _______________________________________________
> >> > Pytables-users mailing list
> >> > Pyt...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >> >
> >> >
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >>
> >> ------------------------------
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Everyone hates slow websites. So do we.
> >> Make your web apps faster with AppDynamics
> >> Download AppDynamics Lite for free today:
> >> http://p.sf.net/sfu/appdyn_d2d_jan
> >>
> >> ------------------------------
> >>
> >> _______________________________________________
> >> Pytables-users mailing list
> >> Pyt...@li...
> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
> >>
> >>
> >> End of Pytables-users Digest, Vol 81, Issue 2
> >> *********************************************
> >>
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Everyone hates slow websites. So do we.
> > Make your web apps faster with AppDynamics
> > Download AppDynamics Lite for free today:
> > http://p.sf.net/sfu/appdyn_d2d_jan
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_jan
>
> ------------------------------
>
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
> End of Pytables-users Digest, Vol 81, Issue 4
> *********************************************
>
|
