Menu
▾
▴
Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 9
|
From: Anthony S. <sc...@gm...> - 2013-02-04 20:44:09
|
Hey David,
I am getting the following error now:
scopatz@ares ~ $ python t.py
10669890 Comparisons
Traceback (most recent call last):
File "t.py", line 61, in <module>
get_hd()
File "t.py", line 54, in get_hd
for (t1, m1, ii), (t2, m2, jj) in combinations(izip(templates, masks,
range(N_irises)), 2):
File "/home/scopatz/.local/lib/python2.7/site-packages/tables/table.py",
line 3308, in __iter__
out=buf_slice)
File "/home/scopatz/.local/lib/python2.7/site-packages/tables/table.py",
line 1807, in read
arr = self._read(start, stop, step, field, out)
File "/home/scopatz/.local/lib/python2.7/site-packages/tables/table.py",
line 1732, in _read
bytes_required))
ValueError: output array size invalid, got 4620 bytes, need 753984000 bytes
And I had to change the phasors line to ths following:
r['phasors'] = np.empty((17, 20*240), complex)
Thanks.
Be Well
Anthony
On Mon, Feb 4, 2013 at 1:41 PM, David Reed <dav...@gm...> wrote:
> I didn't have any luck. I replaced that __iter__ function which led to me
> replacing the read function which lead to me replaceing the _read function
> and I eventually got another error.
>
> Below are 2 functions and my HDF5 Table class declaration. They should be
> self explanatory. I wasn't sure if attachments would go through and this
> is pretty small, so I figured it would be ok just to post. I apologize if
> this is a bit cluttered. I would also appreciate any comments on how I
> assign the results to the matrix D, this does not seem very pythonic at all
> and could use some advice there if its easy. (the ii*jj is just a place
> holder for a more sophisticated measure). Thanks again!
>
> import numpy as np
> import tables as tb
>
> class Iris(tb.IsDescription):
> subject_id = tb.IntCol()
> iris_id = tb.IntCol()
> database = tb.StringCol(5)
> is_left = tb.BoolCol()
> is_flipped = tb.BoolCol()
> templates = tb.BoolCol(shape=(17, 20*480))
> masks1 = tb.BoolCol(shape=(17, 20*480))
> phasors = tb.ComplexCol(itemsize=8, shape=(17, 20*240))
> masks2 = tb.BoolCol(shape=(17, 20*240))
>
>
> def create_hdf5():
> """
> """
> with tb.openFile('test.h5', 'w') as f:
>
> # Create and fill the table of irises",
> irises = f.createTable(f.root, 'irises', Iris, 'Irises',
> filters=tb.Filters(1))
> for ii in range(4620):
>
> r = irises.row
> r['subject_id'] = ii
> r['iris_id'] = 0
> r['database'] = 'test'
> r['is_left'] = True
> r['is_flipped'] = False
> r['templates'] = np.empty((17, 20*480), np.bool8)
> r['masks1'] = np.empty((17, 20*480), np.bool8)
> r['phasors'] = np.empty((17, 20*240)) + 1j*np.empty((17, 20*240))
> r['masks2'] = np.empty((17, 20*240), np.bool8)
> r.append()
>
> irises.flush()
>
> def get_hd():
> """
> """
> from itertools import combinations, izip
> with tb.openFile('test.h5') as f:
> irises = f.root.irises
>
> templates = f.root.irises.cols.templates
> masks = f.root.irises.cols.masks1
>
> N_irises = len(irises)
>
> print '%i Comparisons' % (N_irises*(N_irises - 1)/2)
> D = np.empty((N_irises, N_irises))
> for (t1, m1, ii), (t2, m2, jj) in combinations(izip(templates, masks,
> range(N_irises)), 2):
> D[ii, jj] = ii*jj
>
> np.save('test', D)
>
>
>
>
> On Mon, Feb 4, 2013 at 11:16 AM, <
> pyt...@li...> wrote:
>
>> Send Pytables-users mailing list submissions to
>> pyt...@li...
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> or, via email, send a message with subject or body 'help' to
>> pyt...@li...
>>
>> You can reach the person managing the list at
>> pyt...@li...
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Pytables-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Pytables-users Digest, Vol 81, Issue 7 (Anthony Scopatz)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 4 Feb 2013 10:16:24 -0600
>> From: Anthony Scopatz <sc...@gm...>
>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 7
>> To: Discussion list for PyTables
>> <pyt...@li...>
>> Message-ID:
>> <
>> CAP...@ma...>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> On Mon, Feb 4, 2013 at 9:53 AM, David Reed <dav...@gm...>
>> wrote:
>>
>> > Hi Josh,
>> >
>> > Here is my __iter__ code:
>> >
>> > def __iter__(self):
>> > table = self.table
>> > itemsize = self.dtype.itemsize
>> > nrowsinbuf = table._v_file.params['IO_BUFFER_SIZE'] // itemsize
>> > max_row = len(self)
>> > for start_row in xrange(0, len(self), nrowsinbuf):
>> > end_row = min([start_row + nrowsinbuf, max_row])
>> > buf = table.read(start_row, end_row, 1, field=self.pathname)
>> > for row in buf:
>> > yield row
>> >
>> > It does look different, I will try swapping in the code from github and
>> > see what happens.
>> >
>>
>> Yes, please let us know how that goes! Otherwise send the list both the
>> test data generator script and the script that fails.
>>
>> Be Well
>> Anthony
>>
>>
>> >
>> >
>> > On Mon, Feb 4, 2013 at 9:59 AM, <
>> > pyt...@li...> wrote:
>> >
>> >> Send Pytables-users mailing list submissions to
>> >> pyt...@li...
>> >>
>> >> To subscribe or unsubscribe via the World Wide Web, visit
>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> or, via email, send a message with subject or body 'help' to
>> >> pyt...@li...
>> >>
>> >> You can reach the person managing the list at
>> >> pyt...@li...
>> >>
>> >> When replying, please edit your Subject line so it is more specific
>> >> than "Re: Contents of Pytables-users digest..."
>> >>
>> >>
>> >> Today's Topics:
>> >>
>> >> 1. Re: Pytables-users Digest, Vol 81, Issue 4 (Josh Ayers)
>> >> 2. Re: Pytables-users Digest, Vol 81, Issue 6 (David Reed)
>> >>
>> >>
>> >> ----------------------------------------------------------------------
>> >>
>> >> Message: 1
>> >> Date: Fri, 1 Feb 2013 14:08:47 -0800
>> >> From: Josh Ayers <jos...@gm...>
>> >> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 4
>> >> To: Discussion list for PyTables
>> >> <pyt...@li...>
>> >> Message-ID:
>> >> <CACOB4aPG4NZ6b2a3v=
>> >> 1Ue...@ma...>
>> >> Content-Type: text/plain; charset="iso-8859-1"
>> >>
>> >> David,
>> >>
>> >> You added a custom version of table.Column.__iter__, correct? Could
>> you
>> >> also include that along with the script to reproduce the error?
>> >>
>> >> It seems like the problem may be in the 'nrowsinbuf' calculation - see
>> >> [1]. Each of your rows is 17 x 9600 = 163200 bytes. If you're using
>> the
>> >> default 1MB value for IO_BUFFER_SIZE, it should be reading in rows of 6
>> >> chunks. Instead, it's reading the entire table.
>> >>
>> >> [1]:
>> >>
>> https://github.com/PyTables/PyTables/blob/develop/tables/table.py#L3296
>> >>
>> >>
>> >>
>> >> On Fri, Feb 1, 2013 at 1:50 PM, Anthony Scopatz <sc...@gm...>
>> >> wrote:
>> >>
>> >> >
>> >> >
>> >> > On Fri, Feb 1, 2013 at 3:27 PM, David Reed <dav...@gm...>
>> >> wrote:
>> >> >
>> >> >> at the error:
>> >> >>
>> >> >> result = numpy.empty(shape=nrows, dtype=dtypeField)
>> >> >>
>> >> >> nrows = 4620 and dtypeField is ('bool', (17, 9600))
>> >> >>
>> >> >> I'm not sure what that means as a dtype, but thats what it is.
>> >> >>
>> >> >> Forgive me if I'm being totally naive, but I thought the whole
>> point of
>> >> >> __iter__ with pyttables was to do iteration on the fly, so there is
>> no
>> >> >> preallocation.
>> >> >>
>> >> >
>> >> > Nope you are not being naive at all. That is the point.
>> >> >
>> >> >
>> >> >> If you have any ideas on this I'm all ears.
>> >> >>
>> >> >
>> >> > If you could send a minimal script which reproduces this error, that
>> >> would
>> >> > help a lot.
>> >> >
>> >> > Be Well
>> >> > Anthony
>> >> >
>> >> >
>> >> >>
>> >> >>
>> >> >> Thanks again.
>> >> >>
>> >> >> Dave
>> >> >>
>> >> >>
>> >> >> On Fri, Feb 1, 2013 at 3:45 PM, <
>> >> >> pyt...@li...> wrote:
>> >> >>
>> >> >>> Send Pytables-users mailing list submissions to
>> >> >>> pyt...@li...
>> >> >>>
>> >> >>> To subscribe or unsubscribe via the World Wide Web, visit
>> >> >>>
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> >>> or, via email, send a message with subject or body 'help' to
>> >> >>> pyt...@li...
>> >> >>>
>> >> >>> You can reach the person managing the list at
>> >> >>> pyt...@li...
>> >> >>>
>> >> >>> When replying, please edit your Subject line so it is more specific
>> >> >>> than "Re: Contents of Pytables-users digest..."
>> >> >>>
>> >> >>>
>> >> >>> Today's Topics:
>> >> >>>
>> >> >>> 1. Re: Pytables-users Digest, Vol 81, Issue 2 (Anthony Scopatz)
>> >> >>>
>> >> >>>
>> >> >>>
>> ----------------------------------------------------------------------
>> >> >>>
>> >> >>> Message: 1
>> >> >>> Date: Fri, 1 Feb 2013 14:44:40 -0600
>> >> >>> From: Anthony Scopatz <sc...@gm...>
>> >> >>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue
>> 2
>> >> >>> To: Discussion list for PyTables
>> >> >>> <pyt...@li...>
>> >> >>> Message-ID:
>> >> >>> <
>> >> >>> CAP...@ma...
>> >
>> >> >>> Content-Type: text/plain; charset="iso-8859-1"
>> >> >>>
>> >> >>> On Fri, Feb 1, 2013 at 12:43 PM, David Reed <
>> dav...@gm...>
>> >> >>> wrote:
>> >> >>>
>> >> >>> > Hi Anthony,
>> >> >>> >
>> >> >>> > Thanks for the reply.
>> >> >>> >
>> >> >>> > I honestly don't know how to monitor my Python memory usage, but
>> I'm
>> >> >>> sure
>> >> >>> > that its caused by out of memory.
>> >> >>> >
>> >> >>>
>> >> >>> Well, I would just run top or process monitor or something while
>> >> running
>> >> >>> the python script to see what happens to memory usage as the script
>> >> chugs
>> >> >>> along...
>> >> >>>
>> >> >>>
>> >> >>> > I'm just trying to find out how to fix it. My HDF5 table has
>> 4620
>> >> >>> rows
>> >> >>> > and the column I'm iterating over is a 17x9600 boolean matrix.
>> The
>> >> >>> > __iter__ method is preallocating an array that is this size which
>> >> >>> appears
>> >> >>> > to be root of the error. I was hoping there is a fix somewhere
>> in
>> >> >>> here to
>> >> >>> > not have to do this preallocation.
>> >> >>> >
>> >> >>>
>> >> >>> So a 17x9600 boolean matrix should only be 0.155 MB in space.
>> 4620 of
>> >> >>> these is ~760 MB. If you have 2 GB of memory and you are iterating
>> >> over
>> >> >>> 2
>> >> >>> of these (templates & masks) it is conceivable that you are just
>> >> running
>> >> >>> out of memory. Maybe there is a way that __iter__ could not
>> >> preallocate
>> >> >>> something that is basically a temporary. What is the dtype of the
>> >> >>> templates array?
>> >> >>>
>> >> >>> Be Well
>> >> >>> Anthony
>> >> >>>
>> >> >>>
>> >> >>> >
>> >> >>> > Thanks again.
>> >> >>>
>> >> >>>
>> >> -------------- next part --------------
>> >> An HTML attachment was scrubbed...
>> >>
>> >> ------------------------------
>> >>
>> >> Message: 2
>> >> Date: Mon, 4 Feb 2013 09:58:53 -0500
>> >> From: David Reed <dav...@gm...>
>> >> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 6
>> >> To: pyt...@li...
>> >> Message-ID:
>> >> <CAM6XA7=
>> >> h50...@ma...>
>> >> Content-Type: text/plain; charset="iso-8859-1"
>> >>
>> >> Hi Anthony,
>> >>
>> >> Sorry to just get back to you. I can send a script, should I send a
>> script
>> >> that creates some fake data as well?
>> >>
>> >> -Dave
>> >>
>> >>
>> >> On Fri, Feb 1, 2013 at 4:50 PM, <
>> >> pyt...@li...> wrote:
>> >>
>> >> > Send Pytables-users mailing list submissions to
>> >> > pyt...@li...
>> >> >
>> >> > To subscribe or unsubscribe via the World Wide Web, visit
>> >> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > or, via email, send a message with subject or body 'help' to
>> >> > pyt...@li...
>> >> >
>> >> > You can reach the person managing the list at
>> >> > pyt...@li...
>> >> >
>> >> > When replying, please edit your Subject line so it is more specific
>> >> > than "Re: Contents of Pytables-users digest..."
>> >> >
>> >> >
>> >> > Today's Topics:
>> >> >
>> >> > 1. Re: Pytables-users Digest, Vol 81, Issue 4 (Anthony Scopatz)
>> >> >
>> >> >
>> >> >
>> ----------------------------------------------------------------------
>> >> >
>> >> > Message: 1
>> >> > Date: Fri, 1 Feb 2013 15:50:11 -0600
>> >> > From: Anthony Scopatz <sc...@gm...>
>> >> > Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 4
>> >> > To: Discussion list for PyTables
>> >> > <pyt...@li...>
>> >> > Message-ID:
>> >> > <
>> >> > CAP...@ma...>
>> >> > Content-Type: text/plain; charset="iso-8859-1"
>> >> >
>> >> > On Fri, Feb 1, 2013 at 3:27 PM, David Reed <dav...@gm...>
>> >> wrote:
>> >> >
>> >> > > at the error:
>> >> > >
>> >> > > result = numpy.empty(shape=nrows, dtype=dtypeField)
>> >> > >
>> >> > > nrows = 4620 and dtypeField is ('bool', (17, 9600))
>> >> > >
>> >> > > I'm not sure what that means as a dtype, but thats what it is.
>> >> > >
>> >> > > Forgive me if I'm being totally naive, but I thought the whole
>> point
>> >> of
>> >> > > __iter__ with pyttables was to do iteration on the fly, so there
>> is no
>> >> > > preallocation.
>> >> > >
>> >> >
>> >> > Nope you are not being naive at all. That is the point.
>> >> >
>> >> >
>> >> > > If you have any ideas on this I'm all ears.
>> >> > >
>> >> >
>> >> > If you could send a minimal script which reproduces this error, that
>> >> would
>> >> > help a lot.
>> >> >
>> >> > Be Well
>> >> > Anthony
>> >> >
>> >> >
>> >> > >
>> >> > >
>> >> > > Thanks again.
>> >> > >
>> >> > > Dave
>> >> > >
>> >> > >
>> >> > > On Fri, Feb 1, 2013 at 3:45 PM, <
>> >> > > pyt...@li...> wrote:
>> >> > >
>> >> > >> Send Pytables-users mailing list submissions to
>> >> > >> pyt...@li...
>> >> > >>
>> >> > >> To subscribe or unsubscribe via the World Wide Web, visit
>> >> > >>
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> or, via email, send a message with subject or body 'help' to
>> >> > >> pyt...@li...
>> >> > >>
>> >> > >> You can reach the person managing the list at
>> >> > >> pyt...@li...
>> >> > >>
>> >> > >> When replying, please edit your Subject line so it is more
>> specific
>> >> > >> than "Re: Contents of Pytables-users digest..."
>> >> > >>
>> >> > >>
>> >> > >> Today's Topics:
>> >> > >>
>> >> > >> 1. Re: Pytables-users Digest, Vol 81, Issue 2 (Anthony Scopatz)
>> >> > >>
>> >> > >>
>> >> > >>
>> >> ----------------------------------------------------------------------
>> >> > >>
>> >> > >> Message: 1
>> >> > >> Date: Fri, 1 Feb 2013 14:44:40 -0600
>> >> > >> From: Anthony Scopatz <sc...@gm...>
>> >> > >> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 81,
>> Issue 2
>> >> > >> To: Discussion list for PyTables
>> >> > >> <pyt...@li...>
>> >> > >> Message-ID:
>> >> > >> <
>> >> > >>
>> CAP...@ma...>
>> >> > >> Content-Type: text/plain; charset="iso-8859-1"
>> >> > >>
>> >> > >> On Fri, Feb 1, 2013 at 12:43 PM, David Reed <
>> dav...@gm...>
>> >> > >> wrote:
>> >> > >>
>> >> > >> > Hi Anthony,
>> >> > >> >
>> >> > >> > Thanks for the reply.
>> >> > >> >
>> >> > >> > I honestly don't know how to monitor my Python memory usage, but
>> >> I'm
>> >> > >> sure
>> >> > >> > that its caused by out of memory.
>> >> > >> >
>> >> > >>
>> >> > >> Well, I would just run top or process monitor or something while
>> >> running
>> >> > >> the python script to see what happens to memory usage as the
>> script
>> >> > chugs
>> >> > >> along...
>> >> > >>
>> >> > >>
>> >> > >> > I'm just trying to find out how to fix it. My HDF5 table has
>> 4620
>> >> > rows
>> >> > >> > and the column I'm iterating over is a 17x9600 boolean matrix.
>> The
>> >> > >> > __iter__ method is preallocating an array that is this size
>> which
>> >> > >> appears
>> >> > >> > to be root of the error. I was hoping there is a fix somewhere
>> in
>> >> > here
>> >> > >> to
>> >> > >> > not have to do this preallocation.
>> >> > >> >
>> >> > >>
>> >> > >> So a 17x9600 boolean matrix should only be 0.155 MB in space.
>> 4620
>> >> of
>> >> > >> these is ~760 MB. If you have 2 GB of memory and you are
>> iterating
>> >> > over 2
>> >> > >> of these (templates & masks) it is conceivable that you are just
>> >> running
>> >> > >> out of memory. Maybe there is a way that __iter__ could not
>> >> preallocate
>> >> > >> something that is basically a temporary. What is the dtype of the
>> >> > >> templates array?
>> >> > >>
>> >> > >> Be Well
>> >> > >> Anthony
>> >> > >>
>> >> > >>
>> >> > >> >
>> >> > >> > Thanks again.
>> >> > >> >
>> >> > >> >
>> >> > >> >
>> >> > >> >
>> >> > >> > On Fri, Feb 1, 2013 at 11:12 AM, <
>> >> > >> > pyt...@li...> wrote:
>> >> > >> >
>> >> > >> >> Send Pytables-users mailing list submissions to
>> >> > >> >> pyt...@li...
>> >> > >> >>
>> >> > >> >> To subscribe or unsubscribe via the World Wide Web, visit
>> >> > >> >>
>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> or, via email, send a message with subject or body 'help' to
>> >> > >> >> pyt...@li...
>> >> > >> >>
>> >> > >> >> You can reach the person managing the list at
>> >> > >> >> pyt...@li...
>> >> > >> >>
>> >> > >> >> When replying, please edit your Subject line so it is more
>> >> specific
>> >> > >> >> than "Re: Contents of Pytables-users digest..."
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Today's Topics:
>> >> > >> >>
>> >> > >> >> 1. Re: Pytables-users Digest, Vol 80, Issue 9 (Anthony
>> Scopatz)
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> >
>> ----------------------------------------------------------------------
>> >> > >> >>
>> >> > >> >> Message: 1
>> >> > >> >> Date: Fri, 1 Feb 2013 10:11:47 -0600
>> >> > >> >> From: Anthony Scopatz <sc...@gm...>
>> >> > >> >> Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80,
>> >> Issue 9
>> >> > >> >> To: Discussion list for PyTables
>> >> > >> >> <pyt...@li...>
>> >> > >> >> Message-ID:
>> >> > >> >> <
>> >> > >> >>
>> >> CAP...@ma...>
>> >> > >> >> Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >>
>> >> > >> >> Hi David,
>> >> > >> >>
>> >> > >> >> Sorry, I haven't had a ton of time recently. You seem to be
>> >> getting
>> >> > a
>> >> > >> >> memory error on creating a numpy array. This kind of thing
>> >> typically
>> >> > >> >> happens when you are out of memory. Does this seem to be the
>> case
>> >> > with
>> >> > >> >> you? When this dies, is your memory usage at 100%? If so,
>> this
>> >> > >> algorithm
>> >> > >> >> might require a little tweaking...
>> >> > >> >>
>> >> > >> >> Be Well
>> >> > >> >> Anthony
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> On Fri, Feb 1, 2013 at 6:15 AM, David Reed <
>> >> dav...@gm...>
>> >> > >> >> wrote:
>> >> > >> >>
>> >> > >> >> > I'm still having problems with this one. I can't tell if
>> this
>> >> > >> something
>> >> > >> >> > dumb Im doing with itertools, or if its something in
>> pytables.
>> >> > >> >> >
>> >> > >> >> > Would appreciate any help.
>> >> > >> >> >
>> >> > >> >> > Thanks
>> >> > >> >> >
>> >> > >> >> >
>> >> > >> >> > On Wed, Jan 30, 2013 at 5:00 PM, David Reed <
>> >> > dav...@gm...
>> >> > >> >> >wrote:
>> >> > >> >> >
>> >> > >> >> >> I think I have to reopen this issue. I have been running
>> fine
>> >> for
>> >> > >> >> awhile
>> >> > >> >> >> using the combinations method from itertools, but have
>> recently
>> >> > run
>> >> > >> >> into a
>> >> > >> >> >> memory since I have recently quadrupled the size of the hdf
>> >> file.
>> >> > >> >> >>
>> >> > >> >> >> Here is my code again:
>> >> > >> >> >>
>> >> > >> >> >> from itertools import combinations, izip
>> >> > >> >> >> with tb.openFile(h5_all, 'r') as f:
>> >> > >> >> >> irises = f.root.irises
>> >> > >> >> >>
>> >> > >> >> >> templates = f.root.irises.cols.templates
>> >> > >> >> >> masks = f.root.irises.cols.masks1
>> >> > >> >> >>
>> >> > >> >> >> N_irises = len(irises)
>> >> > >> >> >> index = np.ones((20 * 480), np.bool)
>> >> > >> >> >>
>> >> > >> >> >> print '%i Comparisons' % (N_irises*(N_irises - 1)/2)
>> >> > >> >> >> D = np.empty((N_irises, N_irises))
>> >> > >> >> >> for (t1, m1, ii), (t2, m2, jj) in
>> combinations(izip(templates,
>> >> > >> masks,
>> >> > >> >> >> range(N_irises)), 2):
>> >> > >> >> >> # print ii
>> >> > >> >> >> D[ii, jj] = ham_dist(
>> >> > >> >> >> t1[8, index],
>> >> > >> >> >> t2[:, index],
>> >> > >> >> >> m1[8, index],
>> >> > >> >> >> m2[:, index],
>> >> > >> >> >> )
>> >> > >> >> >>
>> >> > >> >> >> And here is the error:
>> >> > >> >> >>
>> >> > >> >> >> In [10]: get_hd3()
>> >> > >> >> >> 10669890 Comparisons
>> >> > >> >> >>
>> >> > >> >> >>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> ---------------------------------------------------------------------------
>> >> > >> >> >> MemoryError Traceback (most
>> >> recent
>> >> > >> call
>> >> > >> >> >> last)
>> >> > >> >> >> <ipython-input-10-cfb255ce7bd1> in <module>()
>> >> > >> >> >> ----> 1 get_hd3()
>> >> > >> >> >>
>> >> > >> >> >>
>> >> > >> >> >> 118 print '%i Comparisons' %
>> >> > >> (N_irises*(N_irises -
>> >> > >> >> >> 1)/2)
>> >> > >> >> >> 119 D = np.empty((N_irises, N_irises))
>> >> > >> >> >> --> 120 for (t1, m1, ii), (t2, m2, jj) in
>> >> > >> >> >> combinations(izip(temp
>> >> > >> >> >> lates, masks, range(N_irises)), 2):
>> >> > >> >> >> 121 # print ii
>> >> > >> >> >> 122 D[ii, jj] = ham_dist(
>> >> > >> >> >>
>> >> > >> >> >> c:\python27\lib\site-packages\tables\table.pyc in
>> >> __iter__(self)
>> >> > >> >> >> 3274 for start_row in xrange(0, len(self),
>> >> nrowsinbuf):
>> >> > >> >> >> 3275 end_row = min([start_row + nrowsinbuf,
>> >> > max_row])
>> >> > >> >> >> -> 3276 buf = table.read(start_row, end_row, 1,
>> >> > >> >> >> field=self.pathname)
>> >> > >> >> >>
>> >> > >> >> >> 3277 for row in buf:
>> >> > >> >> >> 3278 yield row
>> >> > >> >> >>
>> >> > >> >> >> c:\python27\lib\site-packages\tables\table.pyc in read(self,
>> >> > start,
>> >> > >> >> stop,
>> >> > >> >> >> step,
>> >> > >> >> >> field)
>> >> > >> >> >> 1772 (start, stop, step) =
>> >> > self._processRangeRead(start,
>> >> > >> >> stop,
>> >> > >> >> >> step)
>> >> > >> >> >> 1773
>> >> > >> >> >> -> 1774 arr = self._read(start, stop, step, field)
>> >> > >> >> >> 1775 return internal_to_flavor(arr, self.flavor)
>> >> > >> >> >> 1776
>> >> > >> >> >>
>> >> > >> >> >> c:\python27\lib\site-packages\tables\table.pyc in
>> _read(self,
>> >> > start,
>> >> > >> >> >> stop, step,
>> >> > >> >> >> field)
>> >> > >> >> >> 1719 if field:
>> >> > >> >> >> 1720 # Create a container for the results
>> >> > >> >> >> -> 1721 result = numpy.empty(shape=nrows,
>> >> > >> dtype=dtypeField)
>> >> > >> >> >> 1722 else:
>> >> > >> >> >> 1723 # Recarray case
>> >> > >> >> >>
>> >> > >> >> >> MemoryError:
>> >> > >> >> >> > c:\python27\lib\site-packages\tables\table.py(1721)_read()
>> >> > >> >> >> 1720 # Create a container for the results
>> >> > >> >> >> -> 1721 result = numpy.empty(shape=nrows,
>> >> > >> dtype=dtypeField)
>> >> > >> >> >> 1722 else:
>> >> > >> >> >>
>> >> > >> >> >> Also, if you guys see any performance problems in my code,
>> >> please
>> >> > >> let
>> >> > >> >> me
>> >> > >> >> >> know.
>> >> > >> >> >>
>> >> > >> >> >> Thank you so much for the help.
>> >> > >> >> >>
>> >> > >> >> >> -Dave
>> >> > >> >> >>
>> >> > >> >> >>
>> >> > >> >> >> On Fri, Jan 4, 2013 at 8:57 AM, <
>> >> > >> >> >> pyt...@li...> wrote:
>> >> > >> >> >>
>> >> > >> >> >>> Send Pytables-users mailing list submissions to
>> >> > >> >> >>> pyt...@li...
>> >> > >> >> >>>
>> >> > >> >> >>> To subscribe or unsubscribe via the World Wide Web, visit
>> >> > >> >> >>>
>> >> > >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> or, via email, send a message with subject or body 'help'
>> to
>> >> > >> >> >>> pyt...@li...
>> >> > >> >> >>>
>> >> > >> >> >>> You can reach the person managing the list at
>> >> > >> >> >>> pyt...@li...
>> >> > >> >> >>>
>> >> > >> >> >>> When replying, please edit your Subject line so it is more
>> >> > specific
>> >> > >> >> >>> than "Re: Contents of Pytables-users digest..."
>> >> > >> >> >>>
>> >> > >> >> >>>
>> >> > >> >> >>> Today's Topics:
>> >> > >> >> >>>
>> >> > >> >> >>> 1. Re: Pytables-users Digest, Vol 80, Issue 8 (David
>> Reed)
>> >> > >> >> >>>
>> >> > >> >> >>>
>> >> > >> >> >>>
>> >> > >>
>> >> ----------------------------------------------------------------------
>> >> > >> >> >>>
>> >> > >> >> >>> Message: 1
>> >> > >> >> >>> Date: Fri, 4 Jan 2013 08:56:28 -0500
>> >> > >> >> >>> From: David Reed <dav...@gm...>
>> >> > >> >> >>> Subject: Re: [Pytables-users] Pytables-users Digest, Vol
>> 80,
>> >> > Issue
>> >> > >> 8
>> >> > >> >> >>> To: pyt...@li...
>> >> > >> >> >>> Message-ID:
>> >> > >> >> >>> <
>> >> > >> >> >>>
>> >> > CAM...@ma...
>> >> > >> >
>> >> > >> >> >>> Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>>
>> >> > >> >> >>> I can't thank you guys enough for the help. I was able to
>> add
>> >> > the
>> >> > >> >> >>> __iter__
>> >> > >> >> >>> function to the table.py file and everything seems to be
>> >> working
>> >> > >> >> great!
>> >> > >> >> >>> I'm not quite as fast as I was with iterating right of a
>> >> matrix
>> >> > >> but
>> >> > >> >> >>> pretty
>> >> > >> >> >>> close. I was at 555 comparisons per second, and now im at
>> >> 420.
>> >> > >> >> >>>
>> >> > >> >> >>> I handled the problem I mentioned earlier by doing this,
>> and
>> >> it
>> >> > >> seems
>> >> > >> >> to
>> >> > >> >> >>> work great:
>> >> > >> >> >>>
>> >> > >> >> >>> A = f.root.data.cols.A
>> >> > >> >> >>> B = f.root.data.cols.B
>> >> > >> >> >>>
>> >> > >> >> >>> D = np.empty((len(A), len(A))
>> >> > >> >> >>> for (a1, b1, ii), (a2, b2, jj) in combinations(izip(A, B,
>> >> > >> >> range(len(A))),
>> >> > >> >> >>> 2):
>> >> > >> >> >>> D[ii, jj] = compare(a1, a2, b1, b2)
>> >> > >> >> >>>
>> >> > >> >> >>> Again, thanks a lot.
>> >> > >> >> >>>
>> >> > >> >> >>> -Dave
>> >> > >> >> >>>
>> >> > >> >> >>>
>> >> > >> >> >>> On Thu, Jan 3, 2013 at 6:31 PM, <
>> >> > >> >> >>> pyt...@li...> wrote:
>> >> > >> >> >>>
>> >> > >> >> >>> > Send Pytables-users mailing list submissions to
>> >> > >> >> >>> > pyt...@li...
>> >> > >> >> >>> >
>> >> > >> >> >>> > To subscribe or unsubscribe via the World Wide Web, visit
>> >> > >> >> >>> >
>> >> > >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > or, via email, send a message with subject or body
>> 'help' to
>> >> > >> >> >>> > pyt...@li...
>> >> > >> >> >>> >
>> >> > >> >> >>> > You can reach the person managing the list at
>> >> > >> >> >>> > pyt...@li...
>> >> > >> >> >>> >
>> >> > >> >> >>> > When replying, please edit your Subject line so it is
>> more
>> >> > >> specific
>> >> > >> >> >>> > than "Re: Contents of Pytables-users digest..."
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >> >>> > Today's Topics:
>> >> > >> >> >>> >
>> >> > >> >> >>> > 1. Re: Pytables-users Digest, Vol 80, Issue 3 (Anthony
>> >> > >> Scopatz)
>> >> > >> >> >>> > 2. Re: Pytables-users Digest, Vol 80, Issue 4 (Anthony
>> >> > >> Scopatz)
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >>
>> >> >
>> ----------------------------------------------------------------------
>> >> > >> >> >>> >
>> >> > >> >> >>> > Message: 1
>> >> > >> >> >>> > Date: Thu, 3 Jan 2013 17:26:55 -0600
>> >> > >> >> >>> > From: Anthony Scopatz <sc...@gm...>
>> >> > >> >> >>> > Subject: Re: [Pytables-users] Pytables-users Digest, Vol
>> 80,
>> >> > >> Issue 3
>> >> > >> >> >>> > To: Discussion list for PyTables
>> >> > >> >> >>> > <pyt...@li...>
>> >> > >> >> >>> > Message-ID:
>> >> > >> >> >>> > <CAPk-6T6sz=J5ay_a9YGLPe_yBLGa9c+XgxG0CRNs6fJ=
>> >> > >> >> >>> > Gz...@ma...>
>> >> > >> >> >>> > Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>> >
>> >> > >> >> >>> > On Thu, Jan 3, 2013 at 2:17 PM, David Reed <
>> >> > >> dav...@gm...>
>> >> > >> >> >>> wrote:
>> >> > >> >> >>> >
>> >> > >> >> >>> > > Thanks a lot for the help so far guys!
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > Looking at itertools, I found what I believe to be the
>> >> > perfect
>> >> > >> >> >>> function
>> >> > >> >> >>> > > for what I need, itertools.combinations. This appears
>> to
>> >> be a
>> >> > >> >> valid
>> >> > >> >> >>> > > replacement to the method proposed.
>> >> > >> >> >>> > >
>> >> > >> >> >>> >
>> >> > >> >> >>> > Yes, combinations is awesome!
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > There is a small problem that I didn't mention is that
>> my
>> >> > >> compare
>> >> > >> >> >>> > function
>> >> > >> >> >>> > > actually takes as inputs 2 columns from the table. Like
>> >> so:
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > D = np.empty((N_irises, N_irises))
>> >> > >> >> >>> > > for ii in xrange(N_elements):
>> >> > >> >> >>> > > for jj in xrange(ii+1, N_elements):
>> >> > >> >> >>> > > D[ii, jj] = compare(data['element1'][ii],
>> >> > >> >> >>> > data['element1'][jj],data['element2'][ii],
>> >> > >> >> >>> > > data['element2'][jj])
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > Is there an efficient way of using itertools with this
>> >> > >> structure?
>> >> > >> >> >>> > >
>> >> > >> >> >>> >
>> >> > >> >> >>> > You can always make two other iterators for each column.
>> >> Since
>> >> > >> you
>> >> > >> >> >>> have
>> >> > >> >> >>> > two columns you would have 4 iterators. I am not sure
>> how
>> >> fast
>> >> > >> >> this is
>> >> > >> >> >>> > going to be but I am confident that there is definitely a
>> >> way
>> >> > to
>> >> > >> do
>> >> > >> >> >>> this in
>> >> > >> >> >>> > one for-loop, which is going to be way faster than nested
>> >> > loops.
>> >> > >> >> >>> >
>> >> > >> >> >>> > Be Well
>> >> > >> >> >>> > Anthony
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > On Thu, Jan 3, 2013 at 1:29 PM, <
>> >> > >> >> >>> > > pyt...@li...> wrote:
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >> Send Pytables-users mailing list submissions to
>> >> > >> >> >>> > >> pyt...@li...
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> To subscribe or unsubscribe via the World Wide Web,
>> visit
>> >> > >> >> >>> > >>
>> >> > >> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >> or, via email, send a message with subject or body
>> >> 'help' to
>> >> > >> >> >>> > >> pyt...@li...
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> You can reach the person managing the list at
>> >> > >> >> >>> > >> pyt...@li...
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> When replying, please edit your Subject line so it is
>> >> more
>> >> > >> >> specific
>> >> > >> >> >>> > >> than "Re: Contents of Pytables-users digest..."
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> Today's Topics:
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> 1. Re: Nested Iteration of HDF5 using PyTables
>> (Josh
>> >> > Ayers)
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>>
>> >> > >>
>> >> ----------------------------------------------------------------------
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> Message: 1
>> >> > >> >> >>> > >> Date: Thu, 3 Jan 2013 10:29:33 -0800
>> >> > >> >> >>> > >> From: Josh Ayers <jos...@gm...>
>> >> > >> >> >>> > >> Subject: Re: [Pytables-users] Nested Iteration of HDF5
>> >> using
>> >> > >> >> >>> PyTables
>> >> > >> >> >>> > >> To: Discussion list for PyTables
>> >> > >> >> >>> > >> <pyt...@li...>
>> >> > >> >> >>> > >> Message-ID:
>> >> > >> >> >>> > >> <
>> >> > >> >> >>> > >>
>> >> > >> >>
>> >> CAC...@ma...>
>> >> > >> >> >>> > >> Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> David,
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> The change in issue 27 was only for iteration over a
>> >> > >> >> tables.Column
>> >> > >> >> >>> > >> instance. To use it, tweak Anthony's code as follows.
>> >> This
>> >> > >> will
>> >> > >> >> >>> > iterate
>> >> > >> >> >>> > >> over the "element" column, as in your original
>> example.
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> Note also that this will only work with the
>> development
>> >> > >> version
>> >> > >> >> of
>> >> > >> >> >>> > >> PyTables
>> >> > >> >> >>> > >> available on github. It will be very slow using the
>> >> > released
>> >> > >> >> >>> v2.4.0.
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> from itertools import izip
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> with tb.openFile(...) as f:
>> >> > >> >> >>> > >> data = f.root.data.cols.element
>> >> > >> >> >>> > >> data_i = iter(data)
>> >> > >> >> >>> > >> data_j = iter(data)
>> >> > >> >> >>> > >> data_i.next() # throw the first value away
>> >> > >> >> >>> > >> for i, j in izip(data_i, data_j):
>> >> > >> >> >>> > >> compare(i, j)
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> Hope that helps,
>> >> > >> >> >>> > >> Josh
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> On Thu, Jan 3, 2013 at 9:11 AM, Anthony Scopatz <
>> >> > >> >> sc...@gm...>
>> >> > >> >> >>> > >> wrote:
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> > HI David,
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > Tables and table column iteration have been
>> overhauled
>> >> > >> fairly
>> >> > >> >> >>> recently
>> >> > >> >> >>> > >> > [1]. So you might try creating two iterators,
>> offset
>> >> by
>> >> > >> one,
>> >> > >> >> and
>> >> > >> >> >>> then
>> >> > >> >> >>> > >> > doing the comparison. I am hacking this out super
>> >> quick
>> >> > so
>> >> > >> >> please
>> >> > >> >> >>> > >> forgive
>> >> > >> >> >>> > >> > me:
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > from itertools import izip
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > with tb.openFile(...) as f:
>> >> > >> >> >>> > >> > data = f.root.data
>> >> > >> >> >>> > >> > data_i = iter(data)
>> >> > >> >> >>> > >> > data_j = iter(data)
>> >> > >> >> >>> > >> > data_i.next() # throw the first value away
>> >> > >> >> >>> > >> > for i, j in izip(data_i, data_j):
>> >> > >> >> >>> > >> > compare(i, j)
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > You get the idea ;)
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > Be Well
>> >> > >> >> >>> > >> > Anthony
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > 1. https://github.com/PyTables/PyTables/issues/27
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> > On Thu, Jan 3, 2013 at 9:25 AM, David Reed <
>> >> > >> >> >>> dav...@gm...>
>> >> > >> >> >>> > >> wrote:
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> >> I was hoping someone could help me out here.
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> This is from a post I put up on StackOverflow,
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> I am have a fairly large dataset that I store in
>> HDF5
>> >> and
>> >> > >> >> access
>> >> > >> >> >>> > using
>> >> > >> >> >>> > >> >> PyTables. One operation I need to do on this
>> dataset
>> >> are
>> >> > >> >> pairwise
>> >> > >> >> >>> > >> >> comparisons between each of the elements. This
>> >> requires 2
>> >> > >> >> loops,
>> >> > >> >> >>> one
>> >> > >> >> >>> > to
>> >> > >> >> >>> > >> >> iterate over each element, and an inner loop to
>> >> iterate
>> >> > >> over
>> >> > >> >> >>> every
>> >> > >> >> >>> > >> other
>> >> > >> >> >>> > >> >> element. This operation thus looks at N(N-1)/2
>> >> > comparisons.
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> For fairly small sets I found it to be faster to
>> dump
>> >> the
>> >> > >> >> >>> contents
>> >> > >> >> >>> > >> into a
>> >> > >> >> >>> > >> >> multdimensional numpy array and then do my
>> iteration.
>> >> I
>> >> > run
>> >> > >> >> into
>> >> > >> >> >>> > >> problems
>> >> > >> >> >>> > >> >> with large sets because of memory issues and need
>> to
>> >> > access
>> >> > >> >> each
>> >> > >> >> >>> > >> element of
>> >> > >> >> >>> > >> >> the dataset at run time.
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> Putting the elements into an array gives me about
>> 600
>> >> > >> >> >>> comparisons per
>> >> > >> >> >>> > >> >> second, while operating on hdf5 data itself gives
>> me
>> >> > about
>> >> > >> 300
>> >> > >> >> >>> > >> comparisons
>> >> > >> >> >>> > >> >> per second.
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> Is there a way to speed this process up?
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> Example follows (this is not my real code, just an
>> >> > >> example):
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> *Small Set*:
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> with tb.openFile(h5_file, 'r') as f:
>> >> > >> >> >>> > >> >> data = f.root.data
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> N_elements = len(data)
>> >> > >> >> >>> > >> >> elements = np.empty((N_irises, 1e5))
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> for ii, d in enumerate(data):
>> >> > >> >> >>> > >> >> elements[ii] = data['element']
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> D = np.empty((N_irises, N_irises)) for ii in
>> >> > >> >> xrange(N_elements):
>> >> > >> >> >>> > >> >> for jj in xrange(ii+1, N_elements):
>> >> > >> >> >>> > >> >> D[ii, jj] = compare(elements[ii],
>> >> elements[jj])
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> *Large Set*:
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> with tb.openFile(h5_file, 'r') as f:
>> >> > >> >> >>> > >> >> data = f.root.data
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> N_elements = len(data)
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >> D = np.empty((N_irises, N_irises))
>> >> > >> >> >>> > >> >> for ii in xrange(N_elements):
>> >> > >> >> >>> > >> >> for jj in xrange(ii+1, N_elements):
>> >> > >> >> >>> > >> >> D[ii, jj] =
>> compare(data['element'][ii],
>> >> > >> >> >>> > >> data['element'][jj])
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> >
>> >> > >> >> >>>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> ------------------------------------------------------------------------------
>> >> > >> >> >>> > >> >> Master Visual Studio, SharePoint, SQL, ASP.NET, C#
>> >> 2012,
>> >> > >> >> HTML5,
>> >> > >> >> >>> CSS,
>> >> > >> >> >>> > >> >> MVC, Windows 8 Apps, JavaScript and much more. Keep
>> >> your
>> >> > >> >> skills
>> >> > >> >> >>> > current
>> >> > >> >> >>> > >> >> with LearnDevNow - 3,200 step-by-step video
>> tutorials
>> >> by
>> >> > >> >> >>> Microsoft
>> >> > >> >> >>> > >> >> MVPs and experts. ON SALE this month only -- learn
>> >> more
>> >> > at:
>> >> > >> >> >>> > >> >> http://p.sf.net/sfu/learnmore_122712
>> >> > >> >> >>> > >> >> _______________________________________________
>> >> > >> >> >>> > >> >> Pytables-users mailing list
>> >> > >> >> >>> > >> >> Pyt...@li...
>> >> > >> >> >>> > >> >>
>> >> > >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >>
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >>
>> >> > >> >> >>> >
>> >> > >> >> >>>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> ------------------------------------------------------------------------------
>> >> > >> >> >>> > >> > Master Visual Studio, SharePoint, SQL, ASP.NET, C#
>> >> 2012,
>> >> > >> >> HTML5,
>> >> > >> >> >>> CSS,
>> >> > >> >> >>> > >> > MVC, Windows 8 Apps, JavaScript and much more. Keep
>> >> your
>> >> > >> skills
>> >> > >> >> >>> > current
>> >> > >> >> >>> > >> > with LearnDevNow - 3,200 step-by-step video
>> tutorials
>> >> by
>> >> > >> >> Microsoft
>> >> > >> >> >>> > >> > MVPs and experts. ON SALE this month only -- learn
>> more
>> >> > at:
>> >> > >> >> >>> > >> > http://p.sf.net/sfu/learnmore_122712
>> >> > >> >> >>> > >> > _______________________________________________
>> >> > >> >> >>> > >> > Pytables-users mailing list
>> >> > >> >> >>> > >> > Pyt...@li...
>> >> > >> >> >>> > >> >
>> >> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> >
>> >> > >> >> >>> > >> -------------- next part --------------
>> >> > >> >> >>> > >> An HTML attachment was scrubbed...
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> ------------------------------
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> >
>> >> > >> >> >>>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> ------------------------------------------------------------------------------
>> >> > >> >> >>> > >> Master Visual Studio, SharePoint, SQL, ASP.NET, C#
>> 2012,
>> >> > >> HTML5,
>> >> > >> >> >>> CSS,
>> >> > >> >> >>> > >> MVC, Windows 8 Apps, JavaScript and much more. Keep
>> your
>> >> > >> skills
>> >> > >> >> >>> current
>> >> > >> >> >>> > >> with LearnDevNow - 3,200 step-by-step video tutorials
>> by
>> >> > >> >> Microsoft
>> >> > >> >> >>> > >> MVPs and experts. ON SALE this month only -- learn
>> more
>> >> at:
>> >> > >> >> >>> > >> http://p.sf.net/sfu/learnmore_122712
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> ------------------------------
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> _______________________________________________
>> >> > >> >> >>> > >> Pytables-users mailing list
>> >> > >> >> >>> > >> Pyt...@li...
>> >> > >> >> >>> > >>
>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> End of Pytables-users Digest, Vol 80, Issue 3
>> >> > >> >> >>> > >> *********************************************
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> >
>> >> > >> >> >>>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> ------------------------------------------------------------------------------
>> >> > >> >> >>> > > Master Visual Studio, SharePoint, SQL, ASP.NET, C#
>> 2012,
>> >> > >> HTML5,
>> >> > >> >> CSS,
>> >> > >> >> >>> > > MVC, Windows 8 Apps, JavaScript and much more. Keep
>> your
>> >> > skills
>> >> > >> >> >>> current
>> >> > >> >> >>> > > with LearnDevNow - 3,200 step-by-step video tutorials
>> by
>> >> > >> Microsoft
>> >> > >> >> >>> > > MVPs and experts. ON SALE this month only -- learn more
>> >> at:
>> >> > >> >> >>> > > http://p.sf.net/sfu/learnmore_122712
>> >> > >> >> >>> > > _______________________________________________
>> >> > >> >> >>> > > Pytables-users mailing list
>> >> > >> >> >>> > > Pyt...@li...
>> >> > >> >> >>> > >
>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > -------------- next part --------------
>> >> > >> >> >>> > An HTML attachment was scrubbed...
>> >> > >> >> >>> >
>> >> > >> >> >>> > ------------------------------
>> >> > >> >> >>> >
>> >> > >> >> >>> > Message: 2
>> >> > >> >> >>> > Date: Thu, 3 Jan 2013 17:30:59 -0600
>> >> > >> >> >>> > From: Anthony Scopatz <sc...@gm...>
>> >> > >> >> >>> > Subject: Re: [Pytables-users] Pytables-users Digest, Vol
>> 80,
>> >> > >> Issue 4
>> >> > >> >> >>> > To: Discussion list for PyTables
>> >> > >> >> >>> > <pyt...@li...>
>> >> > >> >> >>> > Message-ID:
>> >> > >> >> >>> > <
>> >> > >> >> >>> >
>> >> > >>
>> CAP...@ma...>
>> >> > >> >> >>> > Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>> >
>> >> > >> >> >>> > Josh is right that you can just edit the code by hand
>> (which
>> >> > >> works
>> >> > >> >> but
>> >> > >> >> >>> > sucks).
>> >> > >> >> >>> >
>> >> > >> >> >>> > However, on Windows -- on the rare occasion when I also
>> >> have to
>> >> > >> >> >>> develop on
>> >> > >> >> >>> > it -- I typically use a distribution that includes a
>> >> compiler,
>> >> > >> >> cython,
>> >> > >> >> >>> > hdf5, and pytables already and then I install my
>> development
>> >> > >> version
>> >> > >> >> >>> from
>> >> > >> >> >>> > github OVER this. I recommend either EPD or Anaconda,
>> >> though
>> >> > >> other
>> >> > >> >> >>> > distributions listed here [1] might also work.
>> >> > >> >> >>> >
>> >> > >> >> >>> > Be well
>> >> > >> >> >>> > Anthony
>> >> > >> >> >>> >
>> >> > >> >> >>> > 1.
>> http://numfocus.org/projects-2/software-distributions/
>> >> > >> >> >>> >
>> >> > >> >> >>> >
>> >> > >> >> >>> > On Thu, Jan 3, 2013 at 3:46 PM, Josh Ayers <
>> >> > jos...@gm...
>> >> > >> >
>> >> > >> >> >>> wrote:
>> >> > >> >> >>> >
>> >> > >> >> >>> > > The change was in pure Python code, so you should be
>> able
>> >> to
>> >> > >> just
>> >> > >> >> >>> paste
>> >> > >> >> >>> > in
>> >> > >> >> >>> > > the changes to your local copy. Start with the
>> >> > >> >> table.Column.__iter__
>> >> > >> >> >>> > > method (lines 3296-3310) here.
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> >
>> >> > >> >> >>>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> https://github.com/PyTables/PyTables/blob/b479ed025f4636f7f4744ac83a89bc947808907c/tables/table.py
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > It needs to be modified slightly because it uses some
>> >> > >> additional
>> >> > >> >> >>> features
>> >> > >> >> >>> > > that aren't available in the released version (the
>> >> > >> out=buf_slice
>> >> > >> >> >>> argument
>> >> > >> >> >>> > > to table.read). The following should work.
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > def __iter__(self):
>> >> > >> >> >>> > > table = self.table
>> >> > >> >> >>> > > itemsize = self.dtype.itemsize
>> >> > >> >> >>> > > nrowsinbuf =
>> >> table._v_file.params['IO_BUFFER_SIZE']
>> >> > //
>> >> > >> >> >>> itemsize
>> >> > >> >> >>> > > max_row = len(self)
>> >> > >> >> >>> > > for start_row in xrange(0, len(self),
>> nrowsinbuf):
>> >> > >> >> >>> > > end_row = min([start_row + nrowsinbuf,
>> >> max_row])
>> >> > >> >> >>> > > buf = table.read(start_row, end_row, 1,
>> >> > >> >> >>> field=self.pathname)
>> >> > >> >> >>> > > for row in buf:
>> >> > >> >> >>> > > yield row
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > I haven't tested this, but I think it will work.
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > Josh
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >
>> >> > >> >> >>> > > On Thu, Jan 3, 2013 at 1:25 PM, David Reed <
>> >> > >> >> dav...@gm...>
>> >> > >> >> >>> > wrote:
>> >> > >> >> >>> > >
>> >> > >> >> >>> > >> I apologize if I'm starting to sound helpless, but I'm
>> >> > forced
>> >> > >> to
>> >> > >> >> >>> work on
>> >> > >> >> >>> > >> Windows 7 at work and have never had luck compiling
>> >> python
>> >> > >> source
>> >> > >> >> >>> > >> successfully. I have had to rely on precompiled
>> binaries
>> >> > and
>> >> > >> now
>> >> > >> >> >>> its
>> >> > >> >> >>> > >> biting me in the butt.
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> Is there any quick fix I can do to improve this
>> iteration
>> >> > >> using
>> >> > >> >> >>> v2.4.0?
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >> On Thu, Jan 3, 2013 at 3:17 PM, <
>> >> > >> >> >>> > >> pyt...@li...> wrote:
>> >> > >> >> >>> > >>
>> >> > >> >> >>> > >>> Send Pytables-users mailing list submissions to
>> >> > >> >> >>> > >>> pyt...@li...
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> To subscribe or unsubscribe via the World Wide Web,
>> >> visit
>> >> > >> >> >>> > >>>
>> >> > >> >> >>>
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >>> or, via email, send a message with subject or body
>> >> 'help'
>> >> > to
>> >> > >> >> >>> > >>> pyt...@li...
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> You can reach the person managing the list at
>> >> > >> >> >>> > >>> pyt...@li...
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> When replying, please edit your Subject line so it is
>> >> more
>> >> > >> >> specific
>> >> > >> >> >>> > >>> than "Re: Contents of Pytables-users digest..."
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> Today's Topics:
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> 1. Re: Pytables-users Digest, Vol 80, Issue 2
>> (David
>> >> > Reed)
>> >> > >> >> >>> > >>> 2. Re: Pytables-users Digest, Vol 80, Issue 3
>> (David
>> >> > Reed)
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>>
>> >> > >> >> >>>
>> >> > >>
>> >> ----------------------------------------------------------------------
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> Message: 1
>> >> > >> >> >>> > >>> Date: Thu, 3 Jan 2013 13:44:29 -0500
>> >> > >> >> >>> > >>> From: David Reed <dav...@gm...>
>> >> > >> >> >>> > >>> Subject: Re: [Pytables-users] Pytables-users Digest,
>> Vol
>> >> > 80,
>> >> > >> >> Issue
>> >> > >> >> >>> 2
>> >> > >> >> >>> > >>> To: pyt...@li...
>> >> > >> >> >>> > >>> Message-ID:
>> >> > >> >> >>> > >>>
>> <CAM6XA7=8ocg5WPD4KLSvLhSw-3BCvq5u7MRxq3Ajd6ha=
>> >> > >> >> >>> > >>> ev...@ma...>
>> >> > >> >> >>> > >>> Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> Thanks Anthony, but unless Im missing something I
>> don't
>> >> > think
>> >> > >> >> that
>> >> > >> >> >>> > method
>> >> > >> >> >>> > >>> will work since this will only be comparing the ith
>> >> element
>> >> > >> with
>> >> > >> >> >>> ith+1
>> >> > >> >> >>> > >>> element. I still need 2 for loops right?
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> Using itertools might speed things up though, I've
>> never
>> >> > used
>> >> > >> >> them
>> >> > >> >> >>> so I
>> >> > >> >> >>> > >>> will give it a shot and let you know how it goes.
>> Looks
>> >> > >> like I
>> >> > >> >> >>> need to
>> >> > >> >> >>> > >>> download the latest release before I do that too.
>> >> Thanks
>> >> > for
>> >> > >> >> the
>> >> > >> >> >>> help.
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> -Dave
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> On Thu, Jan 3, 2013 at 12:12 PM, <
>> >> > >> >> >>> > >>> pyt...@li...> wrote:
>> >> > >> >> >>> > >>>
>> >> > >> >> >>> > >>> > Send Pytables-users mailing list submissions to
>> >> > >> >> >>> > >>> > pyt...@li...
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > To subscribe or unsubscribe via the World Wide Web,
>> >> visit
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>>
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >> > >> >> >>> > >>> > or, via email, send a message with subject or body
>> >> 'help'
>> >> > >> to
>> >> > >> >> >>> > >>> >
>> pyt...@li...
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > You can reach the person managing the list at
>> >> > >> >> >>> > >>> > pyt...@li...
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > When replying, please edit your Subject line so it
>> is
>> >> > more
>> >> > >> >> >>> specific
>> >> > >> >> >>> > >>> > than "Re: Contents of Pytables-users digest..."
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > Today's Topics:
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > 1. Re: Nested Iteration of HDF5 using PyTables
>> >> > (Anthony
>> >> > >> >> >>> Scopatz)
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> >
>> >> > >> >>
>> >> >
>> ----------------------------------------------------------------------
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > Message: 1
>> >> > >> >> >>> > >>> > Date: Thu, 3 Jan 2013 11:11:47 -0600
>> >> > >> >> >>> > >>> > From: Anthony Scopatz <sc...@gm...>
>> >> > >> >> >>> > >>> > Subject: Re: [Pytables-users] Nested Iteration of
>> HDF5
>> >> > >> using
>> >> > >> >> >>> PyTables
>> >> > >> >> >>> > >>> > To: Discussion list for PyTables
>> >> > >> >> >>> > >>> > <pyt...@li...>
>> >> > >> >> >>> > >>> > Message-ID:
>> >> > >> >> >>> > >>> > <CAPk-6T5b=
>> >> > >> >> >>> > >>> >
>> >> 1EG...@ma...
>> >> > >
>> >> > >> >> >>> > >>> > Content-Type: text/plain; charset="iso-8859-1"
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > HI David,
>> >> > >> >> >>> > >>> >
>> >> > >> >> >>> > >>> > Tables and
>
> ...
>
> [Message clipped]
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_jan
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
|
