Re: [Pytables-users] Use of recarrays as representation for Tables in memory

[Josh A. <jos...@gm...>]

There is a big difference in speed when iterating over the rows.  Possibly
that was the reason structured arrays were chosen?  The issue is mentioned
here: http://www.scipy.org/Cookbook/Recarray

In a simple test, I get a difference of about 15x, so it is significant.
Iterating over a recarray with 100,000 rows takes 0.22s, versus 0.014s for
the structured array.  Here's the code.

import time
import numpy as np

dtype = np.format_parser(['i4', 'i4'], [], [])
N = 100000
rec = np.recarray((N, ), dtype=dtype)
struc = np.zeros((N, ), dtype=dtype)

t1 = time.clock()
for row in rec:
    pass
print time.clock() - t1

t1 = time.clock()
for row in struc:
    pass
print time.clock() - t1



On Thu, Jun 28, 2012 at 1:31 PM, Anthony Scopatz <sc...@gm...> wrote:

> On Thu, Jun 28, 2012 at 3:23 PM, Francesc Alted <fa...@py...>wrote:
>
>>  Yes, I think it would make more sense to return a recarray too.
>> However, I remember many time ago (3, 4 years?) that NumPy developers were
>> recommending using structured arrays instead of recarrays.  I don't
>> remember exactly the arguments, but I think that was the reason why the
>> structured arrays were declared the default for reading tables.  But this
>> could be changed, of course...
>>
>
> I remember this too Francesc.  I don't think that this has changed, but I
> forgot the reasons.  Maybe I'll write to the numpy list later tonight,
> unless someone else wants to...
>
>
>>
>> Francesc
>>
>>
>> On 6/28/12 8:25 PM, Anthony Scopatz wrote:
>>
>> Hmmm Ok.  Maybe there needs to be a recarray flavor.
>>
>>  I kind of like just returning a normal ndarray, though I see your
>> argument for returning a recarray.  Maybe some of the other devs can jump
>> in here with an opinion.
>>
>>  Be Well
>> Anthony
>>
>> On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero <al...@mi...>wrote:
>>
>>> I just tested: passing an object of type numpy.core.records.recarray
>>> to the constructor of createTable and then reading back it into memory
>>> via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray.
>>>
>>> Best,
>>>
>>> -á.
>>>
>>>
>>> On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz <sc...@gm...>
>>> wrote:
>>> > Hi Alvaro,
>>> >
>>> > I think if you save the table as a record array, it should return you a
>>> > record array.  Or does it return a structured array?  Have you tried
>>> this?
>>> >
>>> > Be Well
>>> > Anthony
>>> >
>>> > On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero <
>>> al...@mi...>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I've noticed that tables are loaded in memory as structured arrays.
>>> >>
>>> >> It seems that returning recarrays by default would be much in the
>>> >> spirit of the natural naming preferences of PyTables.
>>> >>
>>> >> Is there a reason not to do so?
>>> >>
>>> >> Cheers,
>>> >>
>>> >> Álvaro.
>>> >>
>>> >>
>>> >>
>>> ------------------------------------------------------------------------------
>>> >> Live Security Virtual Conference
>>> >> Exclusive live event will cover all the ways today's security and
>>> >> threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> >> will include endpoint security, mobile security and the latest in
>>> malware
>>> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> >> _______________________________________________
>>> >> Pytables-users mailing list
>>> >> Pyt...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Live Security Virtual Conference
>>> > Exclusive live event will cover all the ways today's security and
>>> > threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> > will include endpoint security, mobile security and the latest in
>>> malware
>>> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> > _______________________________________________
>>> > Pytables-users mailing list
>>> > Pyt...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>>
>> _______________________________________________
>> Pytables-users mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>>
>> --
>> Francesc Alted
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>